Decode ground truth token but get 25% success rate on libero-10
I tried to use FAST on libero_10 dataset, but I finally find out that just encode-decode the ground-truth action using the released tokenizer only gets me 25% success rate. I haven't tried other libero tasks yet. It seems that the tokenizer is a little bit lossy and I can't make it better by changing the action horizon or the vocab size (train a tokenizer on libero).
In you paper I see ~80% success rate on libero dataset. Have you trained a specific tokenizer for libero? If so, what is the hyper-params? Thank you!
Oh, libero spatial gets 100% success rate and libero_object gets 80% success rate. So maybe it is only not good on libero_10? Can you share more if you see the same results? How can we tune the tokenizer to make it less lossy? (scale? vocab size?)
Libero_10 are longer horizon tasks, so it's possible that open-loop replaying actions on these tasks will suffer more from accumulating errors. Note that a closed loop policy can compensate for that, so likely it would get higher success rate than in your experiment (and indeed if you check the openpi repo we list Libero numbers for each benchmark and it gets higher success on Libero_10).
If you want to reduce the tokenization error (ie make reconstructions more precise), the parameter you need to play with is the "scale" parameter -- it controls how lossy the compression is. If you increase scale, the tokenization error will decrease (but at the expense of getting less compressed, ie longer, token sequences for each chunk).
Thank you I will try more!