If you already have a model hosted (finetuned) on AWS SageMaker, you can bring it to Vext for complex LLM pipeline set up.
To add an LLM from your AWS SageMaker:
Go to your project
Add an action by clicking the "+" icon and select "Generate a Response"
Click "Bring Your Own LLM" under the "LLM Type" section
Select "AWS SageMaker" from the drop down menu
And then you will be prompted with additional inputs to set up your AWS SageMaker:
Field | Description | Example |
AWS Access Key ID | A unique identifier associated with your AWS account used to make programmatic requests to AWS. |
|
AWS Secret Access Key | A secret key associated with your Access Key ID, used to securely sign programmatic requests to AWS. |
|
AWS Profile Region Name | The default region name for your AWS profile, specifying where AWS operations will execute by default. |
|
SageMaker Region Name | The region where your SageMaker model is deployed. This may differ from your AWS profile's default region. |
|
Endpoint | The endpoint URL of your deployed SageMaker model, used to send data for inference. |
|
Model Kwargs | Additional keyword arguments specific to your model, used during the model's invocation. Note that this has to be in |
|
Note
Ensure the Model Kwargs JSON adheres to strict JSON syntax: booleans must be lowercase (true/false), and trailing commas at the end of lists or objects are not permitted.
Example:
👍 Good
{"do_sample": true, "top_p": 0.3, "max_new_tokens": 1024, "temperature": 0.8}
👎 Bad
{"do_sample": True, "top_p": 0.3, "max_new_tokens": 1024, "temperature": 0.8,}
Security Measures
We prioritize the security of your AWS access keys and any information you provide. To ensure the highest level of security, we employ Hardware Security Modules (HSMs) that adhere to FIPS 140-2 Level 3 certification standards.
These HSMs are dedicated to performing cryptographic operations and managing keys securely. This certification signifies that our security modules meet rigorous testing standards for physical tamper-resistance, role-based authentication, and secure key management practices.