xAI previews Grok-1.5 and creates a new benchmark called RealWorldQA

Elon Musk’s xAI has revealed Grok-1.5, a multimodal AI model designed to beat competitors in understanding real-world scenarios. 

Following in the footsteps of others, like GPT-4V, the new Grok-1.5 introduces visual processing to analyze anything from documents and diagrams to charts, screenshots, and photographs.

Grok-1.5 also gains ground in text, coding, and math tasks, scoring 50.6% on the MATH benchmark, 90% on the GSM8K benchmark, and 74.1% on the HumanEval benchmark. 

This throws Grok-1.5 right into the LLM heavyweight tier, averaging slightly lower scores than Gemini Pro 1.5, GPT-4, and Claude 3 Opus.

Grok-1.5’s competitive text, math, and coding benchmarks. Source: xAI

Grok-1.5 also offers longer context understanding up to 128K tokens, a 16-fold increase compared to its predecessor, though well behind those touted by Claude 3 Opus and Gemini 1.5 Pro.

A Needle In A Haystack (NIAH) evaluation demonstrated Grok-1.5’s ability to locate embedded text within contexts of up to 128K tokens in length.

However, it’s Grok-1.5’s vision skills that xAI is pushing the hardest.

Demos show Grok-1.5 converting block schemes into Python code, generating bedtime stories inspired by children’s paintings, creating CSV datasets from screenshots, and even “expanding” memes. 

Grok-1.5 tops the leaderboard in some established benchmarks like Mathvista and TextVQA and scores highest in xAI’s newly established benchmark, RealWorldQA.

Grok-1.5’s impressive vision benchmarks. Source: xAI

Under the hood, Grok-1.5 is powered by a custom distributed training framework that enables xAI’s team to prototype ideas and train new architectures at scale with minimal effort.

xAI was founded last year and includes some of the world’s top AI researchers with the ultra-ambitious goal to “Understand the universe.” 

Thus far, we’ve got the witty and outlandish Grok-1 that tells people how to synthesize narcotics and criticizes Musk and Tesla.

Grok is also connected to X’s post database, which, among other unique quirks, has given it quite the following despite not troubling the leaders in pure performance. 

Musk’s xAI project challenges generative AI’s primarily closed-source ecosystem, making its models generally available under true open-source licenses. 

Combined with Meta, which has a similar intent to go against the grain of competitors, xAI’s open thesis might become a thorn in the monetization efforts of OpenAI, Microsoft, Anthropic, and Google.

RealWorldQA

Grok-1.5’s preview also saw xAI reveal the RealWorldQA, a new benchmark consisting of over 700 images, each accompanied by a question and verifiable answer.

The dataset primarily comprises anonymized images captured from vehicles and other real-world situations.

The RealWorldQA dataset is designed to evaluate the spatial understanding capabilities of Grok 1.5 and other multimodal AI models. xAI deemed other benchmarks were lacking in this department. 

The RealWorldQA benchmark dataset seeks to test models’ ability to understand natural scenes. Source: xAI

Grok-1.5 outperforms competitors in RealWorldQA, and it’ll be interesting to see if it catches on.

While it stops short of understanding the universe, Grok-1.5 will take its place as another top-tier model in an ever-increasing lineup.

That also shows how generative AI in its current form is reaching the peaks of its powers – though perhaps not for long. 

The post xAI previews Grok-1.5 and creates a new benchmark called RealWorldQA appeared first on DailyAI.

Source: Read More