Setting Up Your LLM
Ensure that you have at least one LLM configured in your project. This will allow the system to process both text and image inputs.
You need to modify the system prompt for the LLM to handle images. To do this, add a section in your system prompt that explicitly instructs the LLM to interpret the image input. This is done using the file/image system variable.
To add the system variable, type / and select the input_file_1 variable. This will automatically insert the variable into the system prompt, enabling the LLM to handle image files as input.
Before making an API call, it’s a good idea to test the setup in the Vext Playground.
Click on the paperclip icon to upload an image or paste a link to the image.
Once uploaded, the LLM should recognize and process the image according to the instructions in the system prompt.
The returned response from the LLM should demonstrate its ability to interpret and process the image data as instructed.
Calling the API
This section assumes you already know how to integrate your app with your AI Project on Vext. If not, please check out this article first.
Similar to the AI Project API:
curl -XPOST
-H 'Content-Type: application/json'
-H 'Apikey: Api-Key <API_KEY>'
-d '{
"payload": {your_message_here}
}' 'https://payload.vextapp.com/hook/${endpoint_id}/catch/${channel_token}'
If you'd like to include a file from external storage, simply add an additional field to enable image or file uploads.
curl -XPOST
-H 'Content-Type: application/json'
-H 'Apikey: Api-Key <API_KEY>'
-d '{
"payload": {your_message_here},
"input_files": [{
"name": {file_name},
"url": {file_url}
}]
}' 'https://payload.vextapp.com/hook/${endpoint_id}/catch/${channel_token}'
Alternatively, Vext provides a temporary file upload API that allows you to upload files or images and returns a URL upon success. The uploaded file will remain available for 24 hours. You can then take that URL and include it in the API above.
You can check out the temporary file uplaod API documentation here.