Meet ViperGPT: A Python Framework that Combines Vision and Language Models Using Code Generation to Achieve State-of-the-Art Results
The groundbreaking work of Neural Module Networks in prior years aimed to break down jobs into simpler modules. Through training from beginning to finish using modules that were reconfigured for various issues, each module would learn its true purpose and become reusable. Nevertheless, it took a lot of work to use this strategy in the actual world due to several problems. Program development, in particular, needed reinforcement learning from scratch or relied on hand-tuned natural language parsers, making them challenging to optimize. Program creation was severely domain-restricted in each scenario. Training became much more difficult due to learning the perceptual models alongside the program generator, frequently failing to provide the desired modular structure.
As an example let us take some prompts, How many muffins can each child eat for it to be fair? (see Figure 1 (top)) Find the children and the muffins in the image, count how many of each there are, and then decide to divide using the logic that “fair” implies an equitable split. To comprehend the visual environment, it is common for people to compose a mix of many phases. Yet, end-to-end models, which do not naturally use this compositional reasoning, continue to be the dominating strategy in computer vision. Although the discipline has made significant progress on specific tasks like object identification and depth estimation, end-to-end methods to complicated tasks still need to learn to implicitly complete every job during a neural network’s forward run.
