datatypes.eval#

class tonita.datatypes.eval.SubmitEvalRequest(search_requests, email_addresses=None)#

Request for a query evaluation.

Parameters:

  • search_requests (List[SearchRequest]) – The search requests whose results to evaluate.

  • email_addresses (Optional[List[str]]) – An optional list of email addresses to which notifications should be sent about the status of the evaluation.

class tonita.datatypes.eval.SubmitEvalResponse(eval_id)#

Response to a search evaluation submission.

This dataclass contains the evaluation ID, which can be used to check the status of an evaluation and retrieve its results.

Parameters:

eval_id (str) – The ID of this evaluation.

class tonita.datatypes.eval.EvalStatus(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)#

The status of an evaluation.

Parameters:

  • SUBMITTED – The evaluation was successfully submitted.

  • IN_PROGRESS – The evaluation is currently being processed.

  • COMPLETED – The evaluation has been completed.

  • FAILED – The evaluation failed.

  • INVALID – The evaluation is invalid (i.e., does not exist).

class tonita.datatypes.eval.Metrics(precision_at_k)#

Information retrieval metrics for an evaluated query.

Parameters:

precision_at_k (Dict[int, float]) – Precision@k, where each key is an integer for k and each value is the precision value.

class tonita.datatypes.eval.ListingResult(listing_id, rank, score, rating)#

Result for a single listing within a query.

Parameters:

  • listing_id (str) – The ID of the listing.

  • rank (int) – The rank of this listing in the search results given the query.

  • score (float) – The score of this listing in the search results given the query.

  • rating (int) – The evaluation rating of this listing given the query: 1 if relevant and 0 if irrelevant.

class tonita.datatypes.eval.QueryResult(search_request, metrics, listing_results)#

The result for a single query.

Parameters:

  • search_request (SearchRequest) – The search request associated with this query.

  • metrics (Metrics) – Evaluation metrics for this query.

  • listing_results (List[ListingResult]) – The results of the search for this particular query. Each element corresponds to a listing returned in the search results, and elements are sorted in decreasing order of its score.

class tonita.datatypes.eval.RetrieveEvalResponse(status, query_results=None)#

Response for an evaluation retrieval.

This dataclass will return the status of the evaluation and the evaluation results if they are available.

Parameters:

  • status (EvalStatus) – The status of the evaluation.

  • query_results (Optional[List[QueryResult]]) – The evaluation results; one element for each query submitted. This will be None if the status is IN_PROGRESS or FAILED, and will contain results only for a subset of queries submitted if NEAR_COMPLETED.