Public models have more test benchmarks than no public models

[sydddl]

Since my own recent research involves testing the impact of a specific biologically-inspired processing constraint on the brain-like properties of a model, I think Brainscore is a brilliant tool, so I used this framework to compare neural representations in macaques. Given that each experimental condition involves several models with different random seeds, I initially made only one of these models public.

I noticed today that BrainScore has added a number of new benchmarks—for instance, the IT benchmark count has increased from 6 to 13—but these tests appear to have been run only on the public models. I would like to ask whether you plan to run these new tests on the non-public models in the future; if so, roughly how long might that take? Alternatively, if you do not intend to run them automatically, would these additional tests be performed if I were to change the status of my currently non-public models to public? 🤔 I feel that updating the dataset will result in some changes to my results. 😵‍💫

I would also like to add that I wish to conduct a fine-grained analysis of the changes in features across each layer during the model training phase (based on my understanding of the Brainscore project, if a specific layer is specified, only the alignment results for that layer will be returned; if no layer is specified, the system will first calculate all get_layer() results across certain public benchmarks, then select the best one to process the remaining private benchmarks). However, I feel this might place too much of a burden on your resources, so I intend to run all public benchmarks locally. Is there currently a simple and convenient method within the project to download the public benchmarks? Previously, I could only download them one by one as mentioned in the tutorial, and then wait for it to prompt me about network issues or private data.

Thank you very much 🙏

[KartikP]

Hi @sydddl, thank you for your continued engagement with Brain-Score! I would really like to learn more about your work in the future. In terms of what you've asked:

1. New benchmarks not being run on all models
   Brain-Score is currently exploring an alternative approach to scoring models on the new benchmarks. Most models are stale or have been abandoned so we decided to only score new benchmark submissions only on a subset of 100 models. These models represent either the following categories: reference (i.e., models important to the field), top10 models from the 2024 competition, best neural, best behavior, and the remainder of models will be selected from the remaining top models that don't overlap with the above categories. These models are shown on the Explore Page. Since we based this on public models, private models were not scored on the 18 new neural benchmarks.

   We are still working on a mechanism to dynamically update the list of models that will continually be run, but it will generally be based on the model being public.

   Brain-Score tries to continually update its full benchmarking suite to include field-standard benchmarks. The recent additions are from popular datasets like THINGS and Natural Scenes Dataset. To help users maintain their own consistent dashboard, the advanced filter panel includes tools that allow you to select a subset of benchmarks and also view the leaderboard at different timepoints (e.g., from before the new benchmarks were submitted).

   In the short term, I would recommend making models public. You can message me on Slack or GitHub, and have them manually scored on the new benchmarks until the above mechanism is rolled out.

2. Easy way to get all public benchmarks
   Your understanding of Brain-Score is correct and I really appreciate your concern about the compute requirements to perform layer mapping at scale. Currently, there is no method to display all publicly accessible datasets. However, that is a great idea and I will work on that over the next few days. It will likely first start off with a list of public benchmarks on the website. I will subsequently work on a command that users can call to automatically download all public datasets. Additionally, in the near term, we will provide a notebook that users can use to perform an automatic search of model layers to help select the layer mapping.

[sydddl]

Hi @sydddl, thank you for your continued engagement with Brain-Score! I would really like to learn more about your work in the future. In terms of what you've asked:

1. New benchmarks not being run on all models
   Brain-Score is currently exploring an alternative approach to scoring models on the new benchmarks. Most models are stale or have been abandoned so we decided to only score new benchmark submissions only on a subset of 100 models. These models represent either the following categories: reference (i.e., models important to the field), top10 models from the 2024 competition, best neural, best behavior, and the remainder of models will be selected from the remaining top models that don't overlap with the above categories. These models are shown on the Explore Page. Since we based this on public models, private models were not scored on the 18 new neural benchmarks.
   We are still working on a mechanism to dynamically update the list of models that will continually be run, but it will generally be based on the model being public.
   Brain-Score tries to continually update its full benchmarking suite to include field-standard benchmarks. The recent additions are from popular datasets like THINGS and Natural Scenes Dataset. To help users maintain their own consistent dashboard, the advanced filter panel includes tools that allow you to select a subset of benchmarks and also view the leaderboard at different timepoints (e.g., from before the new benchmarks were submitted).
   In the short term, I would recommend making models public. You can message me on Slack or GitHub, and have them manually scored on the new benchmarks until the above mechanism is rolled out.
2. Easy way to get all public benchmarks
   Your understanding of Brain-Score is correct and I really appreciate your concern about the compute requirements to perform layer mapping at scale. Currently, there is no method to display all publicly accessible datasets. However, that is a great idea and I will work on that over the next few days. It will likely first start off with a list of public benchmarks on the website. I will subsequently work on a command that users can call to automatically download all public datasets. Additionally, in the near term, we will provide a notebook that users can use to perform an automatic search of model layers to help select the layer mapping.

Thank you for clarifying that

[sydddl]

Hi @KartikP，After careful consideration, I have decided to resubmit these 16 models, which share the same architecture but were not trained identically (the reason being that, due to my inexperience, I defined the following during submission in accordance with the guidance:
V1 layer1.0
V2 layer3.0
V4 layer3.0
IT layer4.0 mappings
(I had assumed at the time that the JSON file was mandatory, so I first examined which layers were selected in the publicly available ResNet-50 model and filled in the values accordingly).

I subsequently conducted a neural similarity comparison independently on the NSD dataset, selecting the best layer from all available options. My supervisor has asked me to standardise the layer selection method. I apologise for causing your servers to run the same content repeatedly due to my incorrect parameter settings. 🙇‍♂️

I also have a suggestion as a user: currently, there is a ‘public’ option on the user’s dashboard that allows them to decide whether to make their work public. I think it would be useful to add a label so that users can proactively indicate whether a submission is ‘stale or has been abandoned’ (for example, Brainscore is actually quite complex to understand at first time, so some of my submissions are rather a waste of resources). Marking them as abandoned.

As well as the 5GB of User Uploaded Files, could also include a ‘delete’ option, so that during each version update (or via an automatic scan every few weekdays), these files can be released to free up space for models that are to be deleted.

[sydddl]

Perhaps running the new benchmark on all 16 models would be a bit of a drain on resources; after all, you’ve only calculated new scores for 100 models, and many of my models are simply variations with different random seeds.

If you prefer, I could also choose to present only two specific models as individual examples: one with restricted visual development (which has already been tested on the new benchmark and is particularly noteworthy for achieving a top-1 score on the IT task), and the other, which I hope to test on the new benchmark as a corresponding control group.

[KartikP]

Hi @sydddl , I've added your suggestions into our to-do list and will try to address them appropriately.

Feel free to submit the 16 models. If you provide a priority of these models (i.e., the most interesting), I can make sure those ones get scored on all the benchmarks.

[sydddl]

I am extremely grateful. If possible, the order of priority is (resnet50_ewconplus_wzc_2_323, resnet50_st_wzc_seed2_2_323) > (resnet50_ewconplus_wzc_seed4_2_323, resnet50_st_wzc_seed4_2_323) > others. Due to the unique IDs, I’ve added timestamps, which has made the naming a bit clunky. 🙏🥰