top 30 models (across **all** models - 30240 parameter combinations) ```r param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost <int> <num> <int> <num> <char> <char> <char> <num> <int> <char> 1: 17628 -0.7137166 6 0.40 combined perf_min random 133969.98868 7 0.00500143387096774,0.479745967741935 2: 15940 -0.7095018 6 0.20 combined perf_geometric random 211015.95122 7 0.00500143387096774,0.479745967741935 3: 9135 -0.7061135 4 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 4: 6195 -0.7034021 3 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 5: 19312 -0.7018884 7 0.25 combined corr_only random 29826.46078 7 0.00500143387096774,0.479745967741935 6: 19724 -0.7018091 7 0.30 combined perf_mean random 29826.46078 7 0.00500143387096774,0.479745967741935 7: 10866 -0.7017509 4 0.30 combined perf_mean best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 8: 10862 -0.7017393 4 0.30 combined perf_geometric best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 9: 10873 -0.7016459 4 0.30 combined corr_only worst_to_best 1066.57768 5 0.000425496774193549,0.000610483870967742 10: 10870 -0.7015151 4 0.30 combined perf_min best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 11: 18040 -0.7001072 7 0.10 combined perf_geometric random 498969.28141 7 0.00500143387096774,0.479745967741935 12: 13806 -0.6995211 5 0.30 combined perf_mean best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 13: 13802 -0.6995181 5 0.30 combined perf_geometric best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 14: 13810 -0.6994923 5 0.30 combined perf_min best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 15: 13813 -0.6994749 5 0.30 combined corr_only worst_to_best 1066.57768 5 0.000425496774193549,0.000610483870967742 16: 10488 -0.6994024 4 0.25 combined perf_min random 29826.46078 7 0.00500143387096774,0.479745967741935 17: 9652 -0.6991175 4 0.15 combined corr_only random 498549.65927 7 0.00500143387096774,0.479745967741935 18: 10908 -0.6984482 4 0.30 combined perf_min random 29826.46078 7 0.00500143387096774,0.479745967741935 19: 16750 -0.6980950 6 0.30 combined perf_min best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 20: 16746 -0.6980593 6 0.30 combined perf_mean best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 21: 16742 -0.6980588 6 0.30 combined perf_geometric best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 22: 16753 -0.6979782 6 0.30 combined corr_only worst_to_best 1066.57768 5 0.000425496774193549,0.000610483870967742 23: 10904 -0.6973225 4 0.30 combined perf_mean random 30046.28649 7 0.00500143387096774,0.479745967741935 24: 12075 -0.6970813 5 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 25: 7926 -0.6963318 3 0.30 combined perf_mean best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 26: 7922 -0.6963088 3 0.30 combined perf_geometric best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 27: 7933 -0.6959143 3 0.30 combined corr_only worst_to_best 1066.57768 5 0.000425496774193549,0.000610483870967742 28: 7930 -0.6958662 3 0.30 combined perf_min best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 29: 8808 -0.6958639 3 0.40 combined perf_min random 172326.76028 7 0.00500143387096774,0.479745967741935 30: 19690 -0.6957423 7 0.30 combined perf_min best_to_worst 1066.57768 5 0.000425496774193549,0.000610483870967742 param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost best_to_worst random worst_to_best worst2_then_best 13 10 4 3 ``` top 30 models (for cost_bin == 1 or 2 or 3, ie cheapest three bins) ```r param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost <int> <num> <int> <num> <char> <char> <char> <num> <int> <char> 1: 12892 -0.6710269 5 0.20 combined corr_only random 19.16705 1 3.18258064516129e-6,0.0000118854838709677 2: 7424 -0.6675753 3 0.25 combined perf_mean random 19.55709 1 3.18258064516129e-6,0.0000118854838709677 3: 18768 -0.6582458 7 0.20 combined perf_min random 18.51659 1 3.18258064516129e-6,0.0000118854838709677 4: 7433 -0.6567313 3 0.25 combined corr_only worst_to_best 19.28146 1 3.18258064516129e-6,0.0000118854838709677 5: 7853 -0.6567313 3 0.30 combined corr_only worst_to_best 19.28146 1 3.18258064516129e-6,0.0000118854838709677 6: 3086 -0.6544707 2 0.10 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 7: 3506 -0.6544707 2 0.15 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 8: 3924 -0.6544707 2 0.20 weighted perf_mean random 17.06497 1 3.18258064516129e-6,0.0000118854838709677 9: 3926 -0.6544707 2 0.20 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 10: 4346 -0.6544707 2 0.25 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 11: 4766 -0.6544707 2 0.30 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 12: 5186 -0.6544707 2 0.35 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 13: 5606 -0.6544707 2 0.40 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 14: 6026 -0.6544707 3 0.10 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 15: 6446 -0.6544707 3 0.15 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 16: 6866 -0.6544707 3 0.20 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 17: 7286 -0.6544707 3 0.25 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 18: 7706 -0.6544707 3 0.30 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 19: 8126 -0.6544707 3 0.35 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 20: 8546 -0.6544707 3 0.40 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 21: 8966 -0.6544707 4 0.10 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 22: 9386 -0.6544707 4 0.15 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 23: 9806 -0.6544707 4 0.20 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 24: 10226 -0.6544707 4 0.25 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 25: 10646 -0.6544707 4 0.30 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 26: 11066 -0.6544707 4 0.35 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 27: 11486 -0.6544707 4 0.40 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 28: 11906 -0.6544707 5 0.10 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 29: 12326 -0.6544707 5 0.15 weighted perf_mean best_to_worst 17.06497 1 3.18258064516129e-6,0.0000118854838709677 30: 12744 -0.6544707 5 0.20 weighted perf_mean random 17.06497 1 3.18258064516129e-6,0.0000118854838709677 param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost <int> <num> <int> <num> <char> <char> <char> <num> <int> <char> 1: 9135 -0.7061135 4 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 2: 6195 -0.7034021 3 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 3: 12075 -0.6970813 5 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 4: 15015 -0.6862089 6 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 5: 3253 -0.6824805 2 0.10 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 6: 3255 -0.6824805 2 0.10 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 7: 3673 -0.6824805 2 0.15 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 8: 3675 -0.6824805 2 0.15 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 9: 4093 -0.6824805 2 0.20 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 10: 4095 -0.6824805 2 0.20 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 11: 4513 -0.6824805 2 0.25 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 12: 4515 -0.6824805 2 0.25 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 13: 4933 -0.6824805 2 0.30 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 14: 4935 -0.6824805 2 0.30 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 15: 5353 -0.6824805 2 0.35 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 16: 5355 -0.6824805 2 0.35 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 17: 5773 -0.6824805 2 0.40 combined corr_only worst_to_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 18: 5775 -0.6824805 2 0.40 combined corr_only worst2_then_best 50.28532 2 0.0000129181451612903,0.0000331427419354839 19: 17955 -0.6759055 7 0.10 combined corr_only worst2_then_best 73.40478 2 0.0000129181451612903,0.0000331427419354839 20: 9555 -0.6741190 4 0.15 combined corr_only worst2_then_best 64.99771 2 0.0000129181451612903,0.0000331427419354839 21: 21860 -0.6732029 8 0.25 simple perf_geometric random 70.30306 2 0.0000129181451612903,0.0000331427419354839 22: 6615 -0.6729495 3 0.15 combined corr_only worst2_then_best 64.99771 2 0.0000129181451612903,0.0000331427419354839 23: 12495 -0.6721171 5 0.15 combined corr_only worst2_then_best 64.99771 2 0.0000129181451612903,0.0000331427419354839 24: 10395 -0.6718317 4 0.25 combined corr_only worst2_then_best 58.16696 2 0.0000129181451612903,0.0000331427419354839 25: 9975 -0.6716156 4 0.20 combined corr_only worst2_then_best 59.21784 2 0.0000129181451612903,0.0000331427419354839 26: 10815 -0.6715620 4 0.30 combined corr_only worst2_then_best 57.11607 2 0.0000129181451612903,0.0000331427419354839 27: 25232 -0.6709462 9 0.30 simple corr_only random 66.13933 2 0.0000129181451612903,0.0000331427419354839 28: 7455 -0.6708909 3 0.25 combined corr_only worst2_then_best 58.16696 2 0.0000129181451612903,0.0000331427419354839 29: 13755 -0.6707887 5 0.30 combined corr_only worst2_then_best 57.11607 2 0.0000129181451612903,0.0000331427419354839 30: 7035 -0.6706463 3 0.20 combined corr_only worst2_then_best 59.21784 2 0.0000129181451612903,0.0000331427419354839 param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost <int> <num> <int> <num> <char> <char> <char> <num> <int> <char> 1: 22168 -0.6013248 8 0.25 combined perf_min random 107.3645 3 0.0000332201612903226,0.0000909483870967743 2: 3273 -0.5843072 2 0.10 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 3: 3275 -0.5843072 2 0.10 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 4: 3693 -0.5843072 2 0.15 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 5: 3695 -0.5843072 2 0.15 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 6: 4113 -0.5843072 2 0.20 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 7: 4115 -0.5843072 2 0.20 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 8: 4533 -0.5843072 2 0.25 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 9: 4535 -0.5843072 2 0.25 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 10: 4953 -0.5843072 2 0.30 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 11: 4955 -0.5843072 2 0.30 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 12: 5372 -0.5843072 2 0.35 combined corr_only random 144.4606 3 0.0000332201612903226,0.0000909483870967743 13: 5373 -0.5843072 2 0.35 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 14: 5375 -0.5843072 2 0.35 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 15: 5793 -0.5843072 2 0.40 combined corr_only worst_to_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 16: 5795 -0.5843072 2 0.40 combined corr_only worst2_then_best 144.4606 3 0.0000332201612903226,0.0000909483870967743 17: 6776 -0.5803098 3 0.20 simple icc_only random 178.4637 3 0.0000332201612903226,0.0000909483870967743 18: 5922 -0.5794375 3 0.10 simple perf_geometric best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 19: 5926 -0.5794375 3 0.10 simple perf_mean best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 20: 5930 -0.5794375 3 0.10 simple perf_min best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 21: 5933 -0.5794375 3 0.10 simple corr_only worst_to_best 177.7174 3 0.0000332201612903226,0.0000909483870967743 22: 5936 -0.5794375 3 0.10 simple icc_only random 177.7174 3 0.0000332201612903226,0.0000909483870967743 23: 6342 -0.5794375 3 0.15 simple perf_geometric best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 24: 6346 -0.5794375 3 0.15 simple perf_mean best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 25: 6350 -0.5794375 3 0.15 simple perf_min best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 26: 6353 -0.5794375 3 0.15 simple corr_only worst_to_best 177.7174 3 0.0000332201612903226,0.0000909483870967743 27: 6762 -0.5794375 3 0.20 simple perf_geometric best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 28: 6766 -0.5794375 3 0.20 simple perf_mean best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 29: 6770 -0.5794375 3 0.20 simple perf_min best_to_worst 177.7174 3 0.0000332201612903226,0.0000909483870967743 30: 6773 -0.5794375 3 0.20 simple corr_only worst_to_best 177.7174 3 0.0000332201612903226,0.0000909483870967743 param_idx fc_likert_corr max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_per_m_post cost_bin model_cost ```