Deserializing json to Pydantic model

小能豆

Deserializing json to Pydantic model

I have the following Pydantic model:

class OptimizationResponse(BaseModel):
     routes: List[Optional[Route]]
     skippedShipments: Optional[List[SkippedShipment]] = []
     metrics: AggregatedGlobalMetrics

With Route modeled as:

class Route(BaseModel):
    vehicleIndex: Optional[int] = 0
    vehicleStartTime: datetime
    vehicleEndTime: datetime
    visits: List[Visit]
    transitions: List[Transition]
    metrics: AggregatedRouteMetrics
    endLoads: List[EndLoad]
    travelSteps: List[TravelStep]
    vehicleDetour: str

The attribute routes can contain en empty Route, so the json may look like:

{
"routes": [
{},
{non empty route}]...rest of stuff}

And I’m trying to deserialize like:

# Convert the response to JSON format and write it to the file
    json_response = MessageToJson(fleet_routing_response._pb)
    async with aiofiles.open(self.config.get('api', 'response_file'), 'w') as response_file:
        await response_file.write(json_response)
# Deserializing the json
    optimization_response = parse_obj_as(OptimizationResponse, json.loads(json_response))
    return optimization_response

The issue arises when the program is dealing with an empty Route element since is trying to deserialize it even it’s tagged as optional:

routes.0.vehicleStartTime Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.5/v/missing

Otherwise, the deserialization happens without any issue.

And so for the rest of the attributes.

阅读 76

2023-12-16

共1个答案

小能豆

The issue here is that Pydantic, by default, considers Optional fields as required during deserialization if the corresponding key is present in the JSON data, even if the value is None or an empty dictionary.

In your case, when you have an empty Route, it’s still trying to parse it as a non-empty Route and expects the required fields (vehicleStartTime, vehicleEndTime, etc.) to be present.

To handle this, you can use a custom validator for the routes field in your OptimizationResponse model. This validator can check if each element in routes is empty (i.e., an empty dictionary) and replace it with None. Here’s how you can modify your OptimizationResponse class:

from typing import List, Optional
from datetime import datetime
from pydantic import BaseModel, parse_obj_as, validator

class AggregatedGlobalMetrics(BaseModel):
    # Define AggregatedGlobalMetrics fields here
    pass

class AggregatedRouteMetrics(BaseModel):
    # Define AggregatedRouteMetrics fields here
    pass

class EndLoad(BaseModel):
    # Define EndLoad fields here
    pass

class TravelStep(BaseModel):
    # Define TravelStep fields here
    pass

class Visit(BaseModel):
    # Define Visit fields here
    pass

class Transition(BaseModel):
    # Define Transition fields here
    pass

class Route(BaseModel):
    vehicleIndex: Optional[int] = 0
    vehicleStartTime: datetime
    vehicleEndTime: datetime
    visits: List[Visit]
    transitions: List[Transition]
    metrics: AggregatedRouteMetrics
    endLoads: List[EndLoad]
    travelSteps: List[TravelStep]
    vehicleDetour: str

class SkippedShipment(BaseModel):
    # Define SkippedShipment fields here
    pass

class AggregatedGlobalMetrics(BaseModel):
    # Define AggregatedGlobalMetrics fields here
    pass

class OptimizationResponse(BaseModel):
    routes: List[Optional[Route]]
    skippedShipments: Optional[List[SkippedShipment]] = []
    metrics: AggregatedGlobalMetrics

    @validator("routes", pre=True, each_item=True)
    def replace_empty_routes(cls, value):
        # Replace empty Route with None
        if not value or all(v is None for v in value.values()):
            return None
        return value

# Example usage
json_data = {
    "routes": [
        {},
        {"vehicleIndex": 1, "vehicleStartTime": "2022-01-01T00:00:00", "vehicleEndTime": "2022-01-01T01:00:00"},
    ],
    "skippedShipments": [],
    "metrics": {}  # Replace with your actual metrics data
}

optimization_response = OptimizationResponse(**json_data)
print(optimization_response)

With this setup, the replace_empty_routes validator will replace empty Route elements with None, allowing you to use Optional as expected. Adjust the validator based on your specific requirements and the structure of your data.

2023-12-16