Tutorials

Intelligent Search

In organizations where the creation and curation of visual assets plays a critical role (e.g., e-commerce, retail, marketing), management of these assets is crucial. Moreover, as the number of these visual assets grow, discoverability and recommendation solutions become increasingly important to be able to leverage your content.

Using coactive, you can perform intelligent search over your unstructured visual data to understand your content and find the visual assets you are looking for, even when no metadata is available.

For example, you can use coactive to search through millions of images to find those that match a given text description (e.g., "a green dress on a mannequin"), and to recommend products from your catalog that most closely match a user's uploaded image on social media.

Search over unstructured visual data

coactive allows users to easily perform intelligent search at scale through millions of unstructured visual assets. In this notebook we'll demonstrate how you can use coactive's python SDK to perform intelligent search with multimodal queries (i.e. text and/or images).

1. Import dependencies

First, we import any necesary dependencies. Note that you will first need to install the coactive SDK package to your local python environment.

# From python core and other common packages
from PIL import Image
from IPython.display import HTML

# From coactive SDK package
import coactive
from coactive.apis import SimilaritySearchApi

2. Authenticate

Next, we load the authentication and environment variables necessary for calling coactive's APIs. Note that these variables are environment specific and will be provided by Coactive.

from auth_credentials import COACTIVE_HOST, CLIENT_ID, CLIENT_SECRET
from env_variables import CLIENT_EMBEDDING_ID

access_token = f"{CLIENT_ID}:{CLIENT_SECRET}"
configuration = coactive.Configuration(
    host = COACTIVE_HOST,
    access_token = access_token,
)

coactive's intelligent search allows you to submit multimodal queries to perform multimodal semantic search using our SimilaritySearch API. More specifically:

  • Your search query can include text, images or both coactive will then semantically search through your images, video and any associated metadata
  • Below, we highlight these core functions to find specific visual assets within millions of images and videos.

We define a custom function to make it simple to run multiple API requests using the SDK and to visually validate the results using this notebook. This custom function can be found below.

from typing import Optional

from helpers import get_images_df_from_api_response, get_image_from_url, image_formatter

def intelligent_search(
    text_query: Optional[str] = None,
    image_query: Optional[str] = None,
    limit: int = 5) -> None:
    '''
    Example function that displays the resutls of calling the SimilaritySearch API.
    '''
    assert text_query or image_query, 'Need at least one query input.'

    # Display query
    if text_query:
        print(f'Text query: {text_query}')
    if image_query:
        print(f'Image query: {image_query}')
        with Image.open(image_query) as img:
            img.thumbnail(size=(300, 300))
            display(img)

    # Call SimilaritySearchAPI
    with coactive.ApiClient(configuration) as api_client:

        try:
            # For text-only search
            if text_query and not image_query:
                api_response = SimilaritySearchApi(api_client).search_images_by_text_grouped(
                    embedding_id=CLIENT_EMBEDDING_ID,
                    query=text_query,
                    limit=limit)

            # For image-only search
            elif not text_query and image_query:
                api_response = SimilaritySearchApi(api_client).get_similar_images_grouped_semantic(
                    embedding_id=CLIENT_EMBEDDING_ID,
                    file=open(image_query, 'rb'),
                    group_by='',
                    limit=limit)

            # For multimodal search
            elif text_query and image_query:
                api_response = SimilaritySearchApi(api_client).get_similar_images_grouped_semantic(
                    embedding_id=CLIENT_EMBEDDING_ID,
                    file=open(image_query, 'rb'),
                    metadata_value=text_query,
                    group_by='',
                    limit=limit)         

        except coactive.ApiException as e:
            print("Exception when calling SimilaritySearchApi->: %s\n" % e)

    # Display results
    df = get_images_df_from_api_response(api_response)
    display(HTML(df.to_html(formatters={'image': image_formatter}, escape=False, index=False)))