Why Mamba was rejected?

Recently, the International Conference on Learning Representations (ICLR) announced its final decisions for the 2024 conference, drawing significant attention to a particular submission: the Mamba model. This model, initially seen as a major contender against the well-known Transformer architecture for language modeling tasks, was ultimately rejected despite its promising with scores of 8 — 8 — 6 — 3 from reviewers.

Mamba’s rejection raises questions, especially considering its innovative approach as a selective state space model capable of scaling linearly with context length, potentially outperforming the Transformer in certain aspects. Yet, upon closer examination of the reviewers’ feedback, it becomes evident that concerns were primarily about the evaluation methodology.

1. Missing LRA Results: The absence of Long Range Arena (LRA) benchmark results, a standard for evaluating long-sequence models, was a significant gap. The LRA has been a conventional benchmark in similar research, making its omission a notable oversight.

Leave a Comment