Part-3: Scoring and Ranking in an ARD Registry

How the search endpoint decides which agent to return first, why it breaks, and what real world implementations can do differently

Title

Why this post exists

After building the registry in Part 2, I spent some time playing with different queries through the search UI. “Book flight” scored 27. “Book flight from SFO to Tokyo” scored 25. A more specific query scoring lower felt wrong.

I started pulling apart the scoring logic, running edge cases, and talking to colleagues about information retrieval basics. The conversation kept going. Why does this approach break? What should i do differently in real world similar to how Google ranks its web search results? How far can I get, without pulling in a vector store?

This post is a detour from the ARD series. It’s not about the ARD specification, but what we builders encounter in every day life, as we try to integrate new solutions in to our existing environment. The ranking of agentic resources would become a key factor for pushing the autonomous envelope and search quality is a determining factor inside the Agentic registry implementation. If you’re only here for the ARD architecture, skip reading further. If you’ve ever wondered how keyword scoring works and where it falls apart, read on.

The scoring model

The registry uses weighted keyword overlap. It takes the user’s query, tokenize it into number of words, then check how many of those words appear in each field of a catalog entry. Then it weight the fields by importance and Sum the weighted overlaps. That’s the score.

The formula

The score for each field measures how much of the query it covers, scaled by that field’s weight. If 2 out of 3 query words appear in the displayName (weight 30), that field contributes 20 points. Sum across all five fields and cap at 100.

For each field:

field_score = floor(matching_tokens / total_query_tokens × weight)

Where:

matching_tokens = number of query words found in the search text
total_query_tokens = total number of words in the query

Final score = sum of all field scores, capped at 100.

The Problem

The scoring starts with tokenization. The query and every catalog field get split into lowercase words. That’s where the trouble begins.

Both the query and the field values are lowercased and split into individual words:

def tokenize(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

"Flight & Hotel Specialist (A2A)" becomes {"flight", "hotel", "specialist", "a2a"}.

"book flight from SFO to Tokyo" becomes {"book", "flight", "from", "sfo", "to", "tokyo"}.

No stemming or stop word removal. Every token is treated equally.

The problems are already visible in these two examples. The catalog entry contains “flight” but a query with “flights” won’t match it. The query contains “from” and “to” which carry no meaning but count as tokens alongside “flight” and “sfo”. Both of these dilute the score.

Field weights

Each catalog entry has multiple fields: a name, a description, tags, capabilities, and representative queries. A token match in the agent’s name is a much stronger relevance signal than a match buried in a tag. Field weights express that intuition as numbers. Higher weight means a match in that field contributes more to the final score.

Scoring Weights

Example Search Query

Now, Let’s look at an example query and compute the score.

Query: "flight hotel booking" tokens: {"flight", "hotel", "booking"} → 3 tokens

Scoring Partner A’s Flight & Hotel Specialist:

Score Breakdown

Where it breaks

Problem 1: Longer queries score lower

Now, Let’s look at two example queries

"book flight" → we get a score of 27
"book flight from SFO to Tokyo" → we get a score of 25

The second query is more specific and more relevant. It should score higher. But it scores lower because the denominator grew from 2 to 6. The extra words (“from”, “sfo”, “to”, “tokyo”) contribute little overlap but dilute every matching token.

The formula penalizes specificity. A user who provides more context gets a worse match.

Problem 2: No stemming

“flights” and “flight” are different tokens. “booking” and “book” are different tokens. The catalog says “flights” but the user types “flight”. Zero overlap. No credit.

This means the catalog author’s word choice directly determines discoverability. If they wrote “books flights” instead of “search flights” in the description, queries with “book” would match. But they didn’t, so they don’t.

Problem 3: All tokens are weighted equally

In "book flight from SFO to Tokyo":

“flight” is the semantic core of the query
“from” and “to” are grammatical connectors with zero information content
“SFO” and “Tokyo” are parameters, not capability descriptors

The scoring treats all six tokens equally. “from” counts the same as “flight”. This adds noise and reduces the signal-to-noise ratio as queries get more conversational.

Problem 4: No semantic understanding

"I need to pay for my trip" → score ~0 for the Payment Processing Agent
"process payment transaction" → score 41 for the same agent

The first query is what a real user would type. The second is what a developer would type. The scoring only works for developer-style keyword queries because it has no semantic model. It doesn’t know that “pay for my trip” means the same thing as “process payment.”

Problem 5: Score magnitude is meaningless across queries

A score of 48 for “flight hotel booking” (3 tokens) means something different than a score of 25 for “book flight from SFO to Tokyo” (6 tokens). The absolute number depends entirely on query length and vocabulary overlap. You cannot compare scores across different queries. You can only rank within a single query’s result set.

This means the registry can’t answer “is this result good enough?” It can only answer “which result is best among these?”

Solving it incrementally

I tried four approaches in order. Stop word removal (Fix 1) helped with the denominator problem. Stemming (Fix 2) fixed vocabulary mismatch. IDF weighting (Fix 3) was supposed to help with token importance but with only 3 catalog entries, every word got similar IDF scores, so I dropped it. Vector embeddings (Fix 4) solved the semantic gap that the first three couldn’t touch.

Fix 1: Stop word removal

The common English words should be removed before tokenization to improve the quality of scoring algorithm:

STOP_WORDS = {"a", "an", "the", "is", "are", "was", "were", "be",
              "been", "being", "have", "has", "had", "do", "does",
              "did", "will", "would", "could", "should", "may",
              "might", "shall", "can", "need", "to", "of", "in",
              "for", "on", "with", "at", "by", "from", "as", "into",
              "through", "during", "before", "after", "above", "below",
              "between", "out", "off", "over", "under", "again",
              "further", "then", "once", "i", "me", "my", "we", "you",
              "it", "he", "she", "they", "this", "that", "these"}

def tokenize(text):
    tokens = set(re.findall(r"[a-z0-9]+", text.lower()))
    return tokens - STOP_WORDS

Now “book flight from SFO to Tokyo” becomes {"book", "flight", "sfo", "tokyo"} (4 meaningful tokens instead of 6). The denominator shrinks and each real word contributes more.

Fix 2: Stemming

Strip words down to their root so that “flights” and “flight” become the same token before scoring.

flights → flight
booking → book
searches → search
hotels → hotel
processing → process

The simplest approach without external dependencies: a suffix-stripping function that handles common English patterns (-ing, -s, -es, -ed, -tion).

Now “flights” in the catalog matches “flight” in the query. Vocabulary mismatch drops significantly.

Fix 3: IDF weighting (term importance)

IDF gives rare words more influence than common ones. If a word appears in every catalog entry, it tells you nothing. If it appears in only one, it’s a strong signal pointing to that entry.

Not all words carry equal information. “flight” is important. “from” is noise. IDF (Inverse Document Frequency) quantifies this. Now matching “flight” contributes more than matching “agent” because “flight” is more discriminating.

idf(token) = log(total_entries / entries_containing_token)

If “flight” appears in 1 out of 3 entries, its IDF is log(3/1) = 1.1. If “agent” appears in all 3, its IDF is log(3/3) = 0. The scoring formula becomes:

field_score = sum(idf(token) for token in matched_tokens) / sum(idf(token) for token in query_tokens) × weight

Though I looked into IDF technique, i decided not to implement it as i did not find better quality in the results.

Fix 4: Vector embeddings (semantic search)

Vector embeddings convert text into arrays of numbers where semantically similar phrases end up close together in the vector space. “Book a flight” and “reserve plane tickets” share zero words, but their embeddings point in nearly the same direction. The model understands meaning, not just vocabulary.

The idea is to encode each catalog entry as a vector at index time. At query time, encode the query, then rank entries by how close their vectors are to the query vector (cosine similarity). No tokenization or stemming or stop words or IDF (Fix 1, 2 & 3) are needed.

import boto3, json

bedrock = boto3.client("bedrock-runtime")

def embed(text):
    response = bedrock.invoke_model(
        modelId="amazon.titan-embed-text-v2:0",
        body=json.dumps({"inputText": text})
    )
    return json.loads(response["body"].read())["embedding"]

# At index time: embed each entry's combined text
entry_embeddings = [embed(entry_text) for entry in catalog]

# At query time: embed the query, compute cosine similarity
query_embedding = embed(query_text)
scores = [cosine_similarity(query_embedding, e) for e in entry_embeddings]

For example a query “I need to pay for my trip in dollar” which has overlapping keywords returns a high score for the Payment Processing Agent because the embedding model understands semantic equivalence. Keyword overlap technique fails and brings the travel and hotel specialist as top result with a score of 32. Score Comparison With a small catalog (dozens of entries), computing cosine similarity in a loop is fine. When the catalog grows to thousands or tens of thousands of entries, brute-force comparison becomes too slow. That’s when you need a vector store: a database with an index structure optimized for finding the nearest vectors without comparing against every single one.

We’re not building a vector store in this post. The point here is to show where keyword scoring hits its ceiling and what the next step looks like. If there’s interest, that could be a future post.

Conclusion

Stop words and stemming improve keyword scoring, but they don’t bridge the gap between “pay for my trip” and “payment processing.” The words are simply different. Vector embeddings solve this at the root by matching the meaning and not vocabulary.

If I were building this from scratch today, I’d start with vector embeddings. A single Bedrock API call per query gives you semantic understanding without the fragility of token matching. At small scale (dozens to hundreds of catalog entries), cosine similarity in memory is all you need. As the catalog grows to thousands of entries, that’s when you introduce a vector store like OpenSearch with k-NN to handle the index efficiently. But the embedding model stays the same either way.

Next: Part 4, building the agents on Amazon Bedrock AgentCore. One partner agent discovered via ARD, one inhouse agent called directly.

Code for this post: GitHub repo