Amazon Web Services (AWS) has announced vector storage for its S3 cloud object storage – S3 Vectors – in a move it claims will reduce the cost of uploading, storing and querying vectorised data in AI storage by up to 90%.
The aim is to allow customers to cost-effectively store large volumes of vectors in the AWS cloud and search through such indexes to find specific content types. It potentially presents an alternative to more costly vector databases.
Vector data allows for so-called semantic search, where search functionality leverages vector information in metadata to allow users to find similar types of information. Examples might be finding similar scenes in a video file, patterns recording in medical imagery, or collections of documents with related themes.
S3 Vectors introduces a purpose-built AWS bucket type into its S3 object storage and will provide application programming interfaces (APIs) to allow application connectivity to such datastores.
Each Amazon S3 Vectors bucket can support up to 10,000 vector indexes, and each index is capable of storing tens of millions of vectors.
After creating a vector index, customers can also attach metadata as key-value pairs to vectors to filter future queries based on a set of conditions. AWS says S3 Vectors will automatically optimise vector data over time to achieve the best possible price-performance.
S3 Vectors integrates with Amazon Bedrock Knowledge Bases and can be used with Amazon OpenSearch.
Bedrock is AWS’s managed service that allows customers to build generative AI (GenAI) applications, while OpenSearch is a repository and visualisation tool for large volumes of data and to help create retrieval augmented generation (RAG) applications.
S3 Vectors can eliminate the need for provisioning infrastructure for a vector database, according to AWS. That’s presumably because S3 and cloud-based object storage are cheaper to build and run than vector databases.
Object storage is designed to handle large volumes of unstructured data using a flat structure with minimal overheads and allows for efficient retrieval of individual files. Vector databases, meanwhile, are engineered for high-performance similarity search across complex, high-dimensional data. They often rely on specialised indexing methods and hardware acceleration, which can drive up hardware and running costs.
Vector data is a type of high-dimensional data, so called because the number of features or values in a datapoint far exceeds the number of samples or data points collected.
In AI, vectors are used to store data and perform computations on them.
For example, a GenAI request in natural language is processed for word meaning, context, and so on, and then represented in multi-dimensional vector format, upon which mathematical operations can be carried out. This is called vector embedding.
To gain answers to the query, the numerical result of parsing and processing can be compared to already vector-embedded data and an answer supplied.
This means data can represent characteristics that might be found in so-called unstructured data – shapes, colours, and what they may represent when interpreted as a whole, for example.
So far, it appears AWS is the first of the hyperscale cloud providers to introduce vector functionality to its basic object storage offering.
Microsoft Azure provides vector storage and search via Azure Cosmos DB, a vector database. Vector search is possible in Azure using Azure AI search.
Meanwhile, Google Cloud Platform offers vector search via Vertex AI for vector data stored in, for example, GCP’s BigQuery, Cloud SQL or AlloyDB databases.
