Add image input resizing or tiling options

[michaelwasserman]

Prompt API callers may wish to specify whether image inputs are resized (shrunk) or split into tiles.

• Resizing (the default right now) is useful for an overview of an image, e.g. rough descriptions of photos.
• Tiling would be useful for analyzing high-resolution details from a larger image, e.g. OCR from a large document.

API clients could split images themselves if input dimension limits were exposed, per #84

[domenic]

This seems to expose similar issues as #84, where it's dependent on very particular implementation strategies. As such, I think we need to be cautious here about exposing options which might not apply to all models, or apply in future models. So at a minimum, I'd like to see strong developer demand for this.

Secondarily, I'd like to learn more about what exactly "tiling" means. I guess you split up the image and feed in multiple images at once to the model?

A better option here might be to try to expose a simple high-level knob like detail: "low" vs. detail: "high", which is what OpenAI does. That would allow implementations to choose whichever strategy is most appropriate for their architecture.

Have you researched what other model provider APIs do for this, and what level of control they provide?

[michaelwasserman]

So at a minimum, I'd like to see strong developer demand for this.

This request originated from a Chrome team member attempting to demo OCR (optical character recognition) with this API, inputting an image of a full page of text (e.g. rendered from a PDF with page setup for ~8.5x11"). The current Chrome implementation naively scales the image, which makes the text illegible to the model.

Without clear API knobs or knowledge of the underlying implementation, API callers cannot anticipate or specify the level of specificity their image and task requires. Implementers must balance accuracy and processing power, either via compromise or naive preprocessing.

I'd like to learn more about what exactly "tiling" means. I guess you split up the image and feed in multiple images

Yes

A better option here might be to try to expose a simple high-level knob like detail: "low" vs. detail: "high",

A high level knob is a way better idea; and could also likely work for other input types, like audio

Have you researched what other model provider APIs do for this, and what level of control they provide?

No, I haven't, thanks for pointing that out