HAS AI BECOME TOO HUMAN? Researchers At Google AI Find LLMs Can Now Use ML Models And APIs With Just Tool Documentation!

In this era where each day AI seems to be taking over the planet, Large Language Models are growing closer to the human brain more than ever. Researchers at Google have proved that large language models can use undiscovered tools in a zero-shot fashion without prior training by simply presenting an LLM with each tool’s documentation. 

We can think of this entire solution as teaching Audrey, a four-year-old, to ride a bike. Initially, we showed her how to ride a bike and helped her learn (we demonstrate). We showed her how to get on it and ride with training wheels and then without. That is, we showed her all the different scenarios. This solution ultimately deals with the part of how she read about riding a bike in a book (docs), learned about the various functionalities of the bike, and can ride it without any of our help, and she does so quite impressively indeed. She can skid, she can ride with and without training wheels. Seems like our Audrey here is all grown up?

Demonstrations (demos) teach LLMs to use tools by few-shot examples. We may need tons of examples to cover all the tool plans that exist. Documentation (docs) instead teaches LLMs to use tools by describing the functionalities of the tools.

Combinations of including/excluding docs and demos in prompts, as well as varying numbers of demos, were conducted to analyze the results and performance of the model. Experiments were done on six tasks across multiple modalities with various toolsets. The LLM planner used is ChatGPT (gpt-3.5-turbo), and the six tasks were namely: Multi-modal question answering on ScienceQA, Tabular math reasoning on TabMWTabMWP, a maths reasoning dataset, Multi-modal reasoning on NLVRv2, Unseen API usage on a newly collected dataset, Image editing with natural language and Video Tracking. 

They evaluated the model performance, with and without tool documentation, across a varying number of demonstrations (demos) on each dataset. The findings showcase that tool documentation reduces the need for demonstrations. With tool docs, the model seemed to maintain a stable performance even as the number of demonstrations was stripped away. But without tool docs, the model performance showed to be extremely sensitive to the number of demos used.

Through qualitative comparisons, they find that relying on documentation rather than demonstrations provides a more scalable solution to equip large language models with a large number of available tools. Moreover, with tool documentation alone, LLMs are able to comprehend and utilize the most recent vision models to accomplish impressive results on image editing and video tracking tasks by solely using tool docs without any new demos. Researchers have found that although the results are extremely impressive and suggest yet another breakthrough, there is a degradation in performance after the document length exceeds 600 words. 

In turn, this paper addresses not just how LLMs can learn tools through documentation but has shown to replicate the results of popular projects such as ‘Grounded SAM’ and ‘Track Anything’ without additional demonstrations, suggesting a potential for automatic knowledge discovery through tool docs. This gives a new direction in the perspective of tool usage with LLMs entirely and strives to shed light upon the reasoning capabilities of the model.

Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

The post HAS AI BECOME TOO HUMAN? Researchers At Google AI Find LLMs Can Now Use ML Models And APIs With Just Tool Documentation! appeared first on MarkTechPost.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *