concatenate two BPE tokenizer #129
Description
hello. i saw in a paper that they combined two tokenizer, one of them was trained by authors and the second was a pretrained one.
is it possible to do this without having their training data?
if yes, how should we put merges in a correct way?