Tutorial: Vector Search Integration with AWS Bedrock

Overview

This tutorial shows you how to build an AI-powered backend that matches faces by using Go, MongoDB Vector Search, and AWS Bedrock AI models. The application processes an uploaded image, generates a vector embedding, searches for similar faces in MongoDB Atlas, and returns the three closest matches with explanations.

The backend performs the following operations:

Standardizes images to 800x600 JPEG format
Generates 1024-dimensional embeddings by using AWS Bedrock
Queries MongoDB Atlas by using MongoDB Vector Search
Generates similarity explanations by using Claude

How Vector Embeddings Work

An embedding is a vector of floating point numbers that encodes image characteristics. Similar images produce similar vectors, enabling you to find matches by comparing vector proximity in n-dimensional space. The 1024-dimensional embeddings contain values in the range (-1.0, 1.0).

MongoDB Atlas stores pre-computed embeddings for reference images. When a user uploads an image, the backend generates its embedding and uses MongoDB Vector Search to find the closest matches.

Application Flow

The application performs the following steps to process requests:

The frontend captures an image and sends it to the backend as a base64-encoded JSON request.
The backend standardizes the image to 800x600 JPEG format.
AWS Bedrock's amazon.titan-embed-image-v1 model generates an embedding from the image.
MongoDB Vector Search finds the three closest matching embeddings.
AWS Bedrock's anthropic.claude-3-sonnet-20240229-v1:0 model generates a natural language explanation of the similarities.
The backend returns the matched images and explanation to the frontend.

Try the Application

You can test the application at mongodb-celeb-search.com.

Tutorial

This tutorial shows you how to perform the following actions:

Create an HTTP server to handle image processing requests
Standardize uploaded images to a consistent format
Generate vector embeddings by using AWS Bedrock
Query MongoDB Atlas to find similar images
Generate natural language explanations of image similarities

Verify the prerequisites

Before you begin this tutorial, ensure you have the following components:

A MongoDB Atlas account with a configured cluster. To learn how to set up an Atlas cluster, see the MongoDB Get Started guide.
An active AWS account with Bedrock access.
AWS credentials configured in ~/.aws/config and ~/.aws/credentials. To learn more about setting up AWS credentials, see the AWS credential file documentation.
Go installed.

Initialize the module and create the main file

Create a new directory for your project and navigate to it.

To initialize the module, run the following command from your project directory:

go mod init github.com/jdortiz/goai

Then, create a server.go file in the same directory with the basic application structure:

package main
import (
        "context"
        "log"
        "net/http"
        "os"
        "github.com/joho/godotenv"
)
type App struct {
}
func (app App) Start() error {
        const serverAddr string = "0.0.0.0:3001"
        log.Printf("Starting HTTP server: %s\n", serverAddr)
        return http.ListenAndServe(serverAddr, nil)
}
func main() {
        app := App{}
        log.Println(app.Start())
}

Add the image search endpoint

To add a handler method to the App type and register it with the router, add an imageSearch() method to your server.go file, and register it in the Start() method:

 func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
         log.Println("Image search invoked")
 }
 func (app App) Start() error {
         const serverAddr string = "0.0.0.0:3001"
         http.HandleFunc("POST /api/search", app.imageSearch)
         log.Printf("Starting HTTP server: %s\n", serverAddr)
         return http.ListenAndServe(serverAddr, nil)
 }

Start the application by running the following command from your project directory:

go mod tidy
go run server.go

To test the endpoint, run the following command from the command line:

curl -IX POST localhost:3001/api/search

Configure image processing

To define the request structure and add image standardization logic, add the following code to your server.go file:

type CelebMatchRequest struct {
        Image64 string `json:"img"`
}
// Receives a base64 encoded image
func standardizeImage(imageB64 string) (*string, error) {
        // Get the base64 decoder as an io.Reader and use it to decode the image from the data
        b64Decoder := base64.NewDecoder(base64.StdEncoding, strings.NewReader(imageB64))
        origImg, _, err := image.Decode(b64Decoder)
        if err != nil {
                return nil, fmt.Errorf("standardizing image failed: %w", err)
        }
        // Resize to 800x600
        resizedImg := image.NewRGBA(image.Rect(0, 0, 800, 600))
        draw.NearestNeighbor.Scale(resizedImg, resizedImg.Rect, origImg, origImg.Bounds(), draw.Over, nil)
        // Reencode the image to JPEG format with Q=85
        var jpegToSend bytes.Buffer
        if err = jpeg.Encode(&jpegToSend, resizedImg, &jpeg.Options{Quality: 85}); err != nil {
                return nil, fmt.Errorf("standardizing image failed: %w", err)
        }
        // Re-encode to base64
        stdImgB64 := base64.StdEncoding.EncodeToString(jpegToSend.Bytes())
        return &stdImgB64, nil
}

To update the handler to decode and standardize the image, modify the imageSearch() method as shown in the following code:

func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
        // Deserialize request
        var imgReq CelebMatchRequest
        err := json.NewDecoder(r.Body).Decode(&imgReq)
        if err != nil {
                log.Println("ERR: parsing json data", err)
                http.Error(w, err.Error(), http.StatusBadRequest)
                return
        }
        // Split image into metadata and data
        imgParts := strings.Split(imgReq.Image64, ",")
        parts := len(imgParts)
        if parts != 2 {
                log.Printf("ERR: expecting metadata and data. Got %d parts\n", parts)
                http.Error(w, fmt.Sprintf("expecting metadata and data. Got %d parts", parts), http.StatusBadRequest)
                return
        }
        // Decode image from base 64, resize image to 800x600 with Q=85, and re-encode to base64
        stdImage, err := standardizeImage(imgParts[1])
        if err != nil {
                log.Println("ERR:", err)
                http.Error(w, "Error standardizing image", http.StatusInternalServerError)
                return
        }
}

Install the image editing module by running the following command from your project directory:

go get golang.org/x/image/draw

Configure AWS Bedrock

To install the AWS SDK and add configuration information to the App struct, run the following commands from your project directory:

go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/bedrockruntime

Then, add the following AWS configuration fields to the App struct in your server.go file:

type App struct {
        config  *aws.Config
        bedrock *bedrockruntime.Client
}

To initialize the AWS configuration, add the following methods to your server.go file:

func connectToAWS(ctx context.Context) (*aws.Config, error) {
        const dfltRegion string = "us-east-1"
        const credAccount string = "your-account-name"
        cfg, err := config.LoadDefaultConfig(ctx,
                config.WithSharedConfigProfile(credAccount),
                config.WithRegion(dfltRegion),
        )
        return &cfg, err
}
func NewApp(ctx context.Context) (*App, error) {
        cfg, err := connectToAWS(ctx)
        if err != nil {
                log.Println("ERR: Couldn't load default configuration. Have you set up your AWS account?", err)
                return nil, err
        }
        bedrockClient := bedrockruntime.NewFromConfig(*cfg)
        return &App{
                config:  cfg,
                bedrock: bedrockClient,
        }, nil
}

Finally, update the main() method to use the constructor by adding the following code:

func main() {
        ctx := context.Background()
        app, err := NewApp(ctx)
        if err != nil {
                panic(err)
        }
        log.Println(app.Start())
}

Generate embeddings with AWS Bedrock

To define the request structures and create a method to compute embeddings, add the following code to your server.go file:

const titanEmbedImgV1ModelId string = "amazon.titan-embed-image-v1"
const contentTypeJson = "application/json"
type EmbeddingConfig struct {
        OutputEmbeddingLength int `json:"outputEmbeddingLength"`
}
type BedrockRequest struct {
        InputImage      string          `json:"inputImage"`
        EmbeddingConfig EmbeddingConfig `json:"embeddingConfig"`
        InputText       *string         `json:"inputText,omitempty"`
}
func (app App) computeImageEmbedding(ctx context.Context, image string) ([]float64, error) {
        payload := BedrockRequest{
                InputImage: image,
                EmbeddingConfig: EmbeddingConfig{
                        OutputEmbeddingLength: 1024,
                },
                InputText: nil,
        }
        bedrockBody, err := json.Marshal(payload)
        if err != nil {
                return nil, fmt.Errorf("failed to get embedding from bedrock: %w", err)
        }
        bedrockReq := bedrockruntime.InvokeModelInput{
                ModelId:     aws.String(titanEmbedImgV1ModelId),
                Body:        bedrockBody,
                ContentType: aws.String(contentTypeJson),
        }
        embeddingResp, err := app.bedrock.InvokeModel(ctx, &bedrockReq)
        if err != nil {
                return nil, fmt.Errorf("failed to get embedding from bedrock: %w", err)
        }
        result := gjson.GetBytes(embeddingResp.Body, "embedding")
        var embedding []float64
        result.ForEach(func(key, value gjson.Result) bool {
                embedding = append(embedding, value.Float())
                return true
        })
        return embedding, nil
}

Next, install GJSON by running the following command from your project directory:

go get github.com/tidwall/gjson

To update the handler to compute the embedding, modify the imageSearch method as shown in the following code:

 func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
         // ...existing code...
         // Compute the embedding using titan-embed-image-v1
         embedding, err := app.computeImageEmbedding(r.Context(), *stdImage)
         if err != nil {
                 log.Println("ERR:", err)
                 http.Error(w, "Error computing embedding", http.StatusInternalServerError)
                 return
         }
 }

Configure MongoDB Atlas

In your project directory, create an .env file. Add the following code to this file to store your MongoDB Atlas connection URI. Replace the placeholders with your actual cluster and credential information. To learn how to retrieve your connection URI, see the MongoDB Get Started guide.

MONGODB_URI=mongodb+srv://<username>:<password>@<cluster>.mongodb.net/

Next, install the required modules by running the following commands from your project directory:

go get github.com/joho/godotenv
go get go.mongodb.org/mongo-driver/v2

Add a MongoDB client field to the App struct:

 type App struct {
         client  *mongo.Client
         config  *aws.Config
         bedrock *bedrockruntime.Client
 }

To create methods to initialize the MongoDB client, add the following code to your server.go file:

func newDBClient(uri string) (*mongo.Client, error) {
        serverAPI := options.ServerAPI(options.ServerAPIVersion1)
        opts := options.Client().ApplyURI(uri).SetServerAPIOptions(serverAPI)
        client, err := mongo.Connect(opts)
        if err != nil {
                return nil, err
        }
        return client, nil
}
func (app *App) Close() {
        if err := app.client.Disconnect(context.Background()); err != nil {
                panic(err)
        }
}

To update the constructor signature and implementation, modify the NewApp() method as shown in the following code:

func NewApp(ctx context.Context, uri string) (*App, error) {
        cfg, err := connectToAWS(ctx)
        if err != nil {
                log.Println("ERR: Couldn't load default configuration. Have you set up your AWS account?", err)
                return nil, err
        }
        bedrockClient := bedrockruntime.NewFromConfig(*cfg)
        client, err := newDBClient(uri)
        if err != nil {
                log.Println("ERR: connecting to MongoDB cluster:", err)
                return nil, err
        }
        return &App{
                client:  client,
                config:  cfg,
                bedrock: bedrockClient,
        }, nil
}

Finally, to load environment variables, update the main() method as shown in the following code:

func main() {
        var uri string
        err := godotenv.Load()
        if err != nil {
                log.Fatal("Unable to load .env file")
        }
        if uri = os.Getenv("MONGODB_URI"); uri == "" {
                log.Fatal("You must set your 'MONGODB_URI' environment variable. See\n\t https://docs.mongodb.com/drivers/go/current/usage-examples/")
        }
        ctx := context.Background()
        app, err := NewApp(ctx, uri)
        if err != nil {
                panic(err)
        }
        defer func() {
                app.Close()
        }()
        log.Println(app.Start())
}

Query similar images by using MongoDB Vector Search

To create a method to find similar images in MongoDB, add the following code to your server.go file:

func (app App) findSimilarImages(ctx context.Context, embedding []float64) ([]string, error) {
        imgCollection := app.client.Database("celebrity_matcher").Collection("celeb_images")
        vectorSchStage := bson.D{{"$vectorSearch", bson.D{{"index", "vector_index"},
                {"path", "embeddings"},
                {"queryVector", embedding},
                {"numCandidates", 15},
                {"limit", 3}}}}
        projectStage := bson.D{{"$project", bson.D{{"image", 1}}}}
        pipeline := mongo.Pipeline{vectorSchStage, projectStage}
        imgCursor, err := imgCollection.Aggregate(ctx, pipeline)
        if err != nil {
                return nil, fmt.Errorf("failed to get similar images from the database: %w", err)
        }
        similarImgs := []struct {
                Id    bson.ObjectID `bson:"_id,omitempty"`
                Image string        `bson:"image"`
        }{}
        if err = imgCursor.All(ctx, &similarImgs); err != nil {
                return nil, fmt.Errorf("failed to get similar images from the database: %w", err)
        }
        var images []string
        var stdImage *string
        for _, item := range similarImgs {
                stdImage, err = standardizeImage(item.Image)
                if err != nil {
                        return nil, fmt.Errorf("failed to standardize similar images: %w", err)
                }
                images = append(images, *stdImage)
        }
        return images, nil
}

To update the handler to find similar images, modify the imageSearch method as follows:

 func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
         // ...existing code...
         // Find similar images using vector search in MongoDB
         images, err := app.findSimilarImages(r.Context(), embedding)
         if err != nil {
                 log.Println("ERR:", err)
                 http.Error(w, "Error getting similar images", http.StatusInternalServerError)
                 return
         }
 }

Generate similarity explanations with Claude

To define the Claude request structures and create a method to generate explanations, add the following code to your server.go file:

const claude3SonnetV1ModelId string = "anthropic.claude-3-sonnet-20240229-v1:0"
type ClaudeBodyMsgSource struct {
        Type      string  `json:"type"`
        MediaType *string `json:"media_type,omitempty"`
        Data      *string `json:"data,omitempty"`
}
type ClaudeBodyMsgContent struct {
        Type   string               `json:"type"`
        Source *ClaudeBodyMsgSource `json:"source,omitempty"`
        Text   *string              `json:"text,omitempty"`
}
type ClaudeBodyMsg struct {
        Role    string                 `json:"role"`
        Content []ClaudeBodyMsgContent `json:"content"`
}
type ClaudeBody struct {
        AnthropicVersion string          `json:"anthropic_version"`
        MaxTokens        int             `json:"max_tokens"`
        System           string          `json:"system"`
        Messages         []ClaudeBodyMsg `json:"messages"`
}
func (app App) getImageSimilaritiesDescription(ctx context.Context, imgB64 string, similarImgB64 []string) (*string, error) {
        const mediaTypeImage = "image/jpeg"
        prompt := "Please let the user know how their first image is similar to the other 3 and which one is the most similar?"
        payload := ClaudeBody{
                AnthropicVersion: "bedrock-2023-05-31",
                MaxTokens:        1000,
                System:           "Please act as face comparison analyzer.",
                Messages: []ClaudeBodyMsg{
                        {
                                Role: "user",
                                Content: []ClaudeBodyMsgContent{
                                        {
                                                Type: "image",
                                                Source: &ClaudeBodyMsgSource{
                                                        Type:      "base64",
                                                        MediaType: aws.String(mediaTypeImage),
                                                        Data:      &imgB64,
                                                },
                                        },
                                        {
                                                Type: "image",
                                                Source: &ClaudeBodyMsgSource{
                                                        Type:      "base64",
                                                        MediaType: aws.String(mediaTypeImage),
                                                        Data:      &similarImgB64[0],
                                                },
                                        },
                                        {
                                                Type: "image",
                                                Source: &ClaudeBodyMsgSource{
                                                        Type:      "base64",
                                                        MediaType: aws.String(mediaTypeImage),
                                                        Data:      &similarImgB64[1],
                                                },
                                        },
                                        {
                                                Type: "image",
                                                Source: &ClaudeBodyMsgSource{
                                                        Type:      "base64",
                                                        MediaType: aws.String(mediaTypeImage),
                                                        Data:      &similarImgB64[2],
                                                },
                                        },
                                        {
                                                Type: "text",
                                                Text: &prompt,
                                        },
                                },
                        },
                },
        }
        bedrockBody, err := json.Marshal(payload)
        if err != nil {
                return nil, fmt.Errorf("failed to get embedding from bedrock: %w", err)
        }
        bedrockReq := bedrockruntime.InvokeModelInput{
                ModelId:     aws.String(claude3SonnetV1ModelId),
                Body:        bedrockBody,
                ContentType: aws.String(contentTypeJson),
                Accept:      aws.String(contentTypeJson),
        }
        bedrockResp, err := app.bedrock.InvokeModel(ctx, &bedrockReq)
        if err != nil {
                return nil, fmt.Errorf("failed to get embedding from bedrock: %w", err)
        }
        description := gjson.GetBytes(bedrockResp.Body, "content.0.text").String()
        return &description, nil
}

To update the handler to generate the description, modify the imageSearch method as shown in the following code:

 func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
         // ...existing code...
         description, err := app.getImageSimilaritiesDescription(r.Context(), *stdImage, images)
         if err != nil {
                 log.Println("ERR: failed to describe similarities with images", err)
                 http.Error(w, "Error describing similarities with images", http.StatusInternalServerError)
                 return
         }
 }

Return the results

To define the response structure and complete the handler, add the following code to your server.go file:

type CelebMatchResponse struct {
        Description string   `json:"description"`
        Images      []string `json:"images"`
}
func (app App) imageSearch(w http.ResponseWriter, r *http.Request) {
        // ...existing code...
        response := CelebMatchResponse{
                Description: *description,
                Images:      images,
        }
        jData, err := json.Marshal(response)
        if err != nil {
                log.Println("error serializing json", err)
                http.Error(w, "error serializing json", http.StatusInternalServerError)
                return
        }
        w.Header().Set("Content-Type", contentTypeJson)
        w.Header().Set("Content-Length", strconv.Itoa(len(jData)))
        w.WriteHeader(http.StatusOK)
        w.Write(jData)
}

Start the application once more by running the following command from your project directory:

go mod tidy
go run server.go

Navigate to http://localhost:3001 in your web browser and test the API by sending a POST request to /api/search with a base64-encoded image.

Additional Resources

To learn more about MongoDB Vector Search, see the MongoDB Vector Search documentation in the MongoDB Server manual.

To learn more about AWS Bedrock, see the AWS Bedrock documentation.

Back

TLS Security Protocol

Release Notes