Recording and reviewing URL replacements

[xhluca]

This code is used to replace the pattern __SHOPPING__ into the correct URL. Is there a way to inspect the replacement post-hoc (after inference) to ensure that tasks were correctly inspected?

Here's the replacement code:

BrowserGym/browsergym/webarena/src/browsergym/webarena/task.py

Lines 56 to 63 in 66cf402

	for pattern, url_key in {
	"__GITLAB__": "gitlab",
	"__REDDIT__": "reddit",
	"__SHOPPING__": "shopping",
	"__SHOPPING_ADMIN__": "shopping_admin",
	"__WIKIPEDIA__": "wikipedia",
	"__MAP__": "map",
	}.items():

I have noticed that in many cases a reward of 0 is assigned even though the correct URL was navigated at certain point

[gasse]

Hi @xhluca , I don't think you will be able to see that post-hoc in the traces. The easiest way to check would be to set yourself in debug mode, place a few breakpoints in the code and inspect the self.config variable. You could also go down to webarena's evaluator function and try to understand at which point the URLs are checked.