OpenAI’s GPT-4V storms out of the gate: Users go bananas

OpenAI’s newest release, the multimodal GPT-4V (”V” is for Vision, get it?), which allows users to use images in their prompts, has stormed into the market, grabbing more attention than any previous release other than the game-changing ChatGPT. As planned, OpenAI has effectively pre-empted all that buzz about the suuuper-cool upcoming new multimodal Google LLM, “Genesis.”

The new use cases are insane - see this demo of 35 GPT-4V use cases

OpenAI is still top dog in the AI-to-consumer market, and is not about to let up. They keep releasing and shipping, while Google keeps talking about how good it’s gonna be, Real Soon Now.

OpenAI CEO Sam Altman was formerly the CEO of premiere startup incubator Y Combinator, which taught fledgling startups how to scale, and how to monetize. He demonstrates over and over that he is well versed in how to do both.

Anthropic in talks to raise another $2 billion, just a week after the $4 billion Amazon deal

AI mania makes for strange bedfellows, and Anthropic, creator of the Claude LLM, is bedhopping with the best of them, Just 6 months ago, Google invested $300+ million into the “We’re Not OpenAI” startup, and Anthropic called Google its “preferred cloud provider.” Then last week, Amazon promised up to $4 billion to the company, and all of a sudden Amazon was its “primary cloud provider.” Was Google cheesed off??? Maybe so, but it apparently hasn’t kept Google from competing to participate in an even newer $2 billion round of funding, which will value the company at $20-$30 billion, 5x its value in April. Anthropic clearly knows that when it’s raining money, you grab a bucket.

Anthropic plays the field, grabs beaucoup cash

Andrej Karpathy says LLMs are the new OS

AI legend Andrej Karpathy has seen the future, and in that future, LLMs like ChatGPT are central coordinators of an integrated collection of specialized tools. In his view, we have been overly dazzled by LLMs’ wide array of abilities, and have mistakenly been using them like a general purpose tool - a kind of Leatherman of the mind. But in fact, no one uses a multi-tool for production work.

LLMs are very good at just a few tasks, and these are all in general some species of translation. Translating input text into pictures, or into computer code, or into a summarized version of the input, or… you get the picture. This makes them perfect as a central interface between humans and the app, or between different special purpose tools.

So, just as in a conventional computer architecture where an operating system coordinates the special-purpose units such as the CPU, the memory, etc., LLMs may emerge as the OS of intelligent systems. Foundational models from OpenAI, Google, Meta, etc. may be analogous to Windows, IOS, Linux, and so on. Each will have native apps, plus an app store.

He summarizes this vision thusly:

TLDR looking at LLMs as chatbots is the same as looking at early computers as calculators. We're seeing an emergence of a whole new computing paradigm, and it is very early.

Andrej Karpathy has seen the future of AI

Bing Chat now uses DALL-E v3 for images, and wow!

For months, the best way to get GPT-4 access for free was to use Microsoft’s Bing search app in Chat mode. This included access to DALL-E, a capable if somewhat pedestrian text-to-image generator. Now Bing has upgraded to GPT-4V (see this week’s Top Story for the implications of that) and this includes access to the latest upgrade to DALL-E. The difference is immediately apparent - the images are more sophisticated, you have more control of the final product through your prompting, and it actually handles text quite well.

All of the images in this issue of the newsletter have been generated by Bing Chat.

AI model says volcanoes, not an asteroid, doomed the dinosaurs

Scientists have built a special AI computer to simulate climate conditions around the time the dinosaurs died out, and it is favoring a much larger role for erupting volcanoes than was formerly thought.

AI fingers volcanoes in death of the dinosaurs

AI predicts earthquake aftershocks better than SOTA models

In this Nature article, scientists report that an AI model trained on reams of data from past earthquakes is now better at predicting aftershocks than the best conventional models.

AI predicts earthquake aftershocks

How Open Source LLMs can catch up to GPT-4V and Gemini

Bindu Reddy, CEO of AbacusAI, tweets about the David-and-Goliath race between closed-source behemoths like OpenAI and Google, and the active and creative community of open source model developers found on GitHub and HuggingFace. Dismissing the naysayers, Reddy lays out a roadmap for how open source developers can not only compete, but win. 

Bindu Reddy on how Open Source can beat OpenAI

UK researchers model farm robots on ant brain

Agricultural robots are being used increasingly on farms around the world. Researchers in the UK are modeling the navigation system of one such series of agri-bots on the brain of an ant. The scientists point out that ants navigate complex real world environments with a brain the size of a pinhead, so there are potentially a lot of lessons there on efficiency and miniaturization.

UK scientists model farm robots on ant brain

GPT-4 scores 90% on USMLE soft skills inventory, better than most physicians

In this Nature article, researchers report that GPT-4 scored 90% on the USMLE soft skills questions, which evaluate empathy and communication skills. This adds to a growing body of evidence that the chatty, friendly, helpful chatbots can be perceived as more empathic than actual physicians.

GPT-4 scores high on empathy in USMLE

NYU Langone Health system holds AI “Prompt-A-Thon”

NYU Langone Health, one of the nation’s largest health systems, held an AI “prompt-a-thon” in August. During this event, teams of clinicians, educators, and researchers worked on AI-powered solutions to health care problem areas, using real world, de-identified data. The goal was to engage the health system community in thinking about how AI could help improve care and make it more efficient and effective. By all accounts, the event was a signal success.

NYU Langone prompt-a-thon

Geisinger has one of nation’s first Medical Director of AI

Dr. Aalpen Patel, Geisinger’s Medical Director of AI, is one of the first physicians to hold such a position in the US. In this interview, he talks about his views on the future of AI in medical care.

Geisinger’s Medical Director of AI on the future of AI in health care