DCXTribeDCXTribe
Back to library
AI
tutorial

Analyze Images and Text Together with Multimodal Vision

Automate visual analysis tasks using Gemini's multimodal Pro Vision API.

Use Gemini Pro Vision to process both image and text inputs simultaneously for complex visual reasoning and data extraction.

Google Gemini

The Scenario

You need to extract line items from a photographed receipt or get a code snippet based on a screenshot of a website design.

Before & after

The old way

A worker would manually transcribe text from a screenshot or describe an image for a colleague, which can take 5-10 minutes depending on complexity.

With AI

With the Gemini Pro Vision API, you can upload the image and the prompt together, receiving a structured explanation of the image in just 5-10 seconds.

The Prompt

You are using the Google GenAI SDK. Use this structure to analyze an image:

const model = genAI.getGenerativeModel({ model: "gemini-pro-vision" });
const prompt = "[WHAT_DO_YOU_WANT_TO_KNOW_ABOUT_THE_IMAGE]";
const imageParts = [
  fileToGenerativePart("input_image.jpg", "image/jpeg"),
];

const result = await model.generateContent([prompt, ...imageParts]);
const response = await result.response;
console.log(response.text());

This allows for 'visual' automation, such as reading receipts, identifying UI components in a design, or describing hardware errors from a photo.

Source

Release notes  |  Gemini API  |  Google AI for Developers
"explore the Gemini Pro Vision API (Multimodal) Generate text from image and text input."