Prep the dish: Grease a 9x13 inch baking dish with butter.
Set aside.
Cube the bread: Cut the brioche or challah into 1-inch cubes. If the bread is very fresh, spread the cubes on a sheet pan and let them sit out for 30 to 60 minutes to dry a bit.
Layer the casserole: Add half the bread cubes to the dish. Scatter half the blueberries over the bread.
Top with the remaining bread and then the remaining blueberries.
Make the custard: In a large bowl, whisk together eggs, milk, granulated sugar, brown sugar, vanilla, cinnamon, nutmeg, salt, melted butter, and lemon zest if using. Whisk until smooth and well combined.
Soak the bread: Pour the custard evenly over the bread and berries, pressing gently with the back of a spatula so the top layer starts to absorb the liquid. Cover tightly with foil or wrap.
Chill overnight: Refrigerate for at least 6 hours and up to 18 hours.
The longer soak helps the center become custardy and tender.
Preheat and prep: In the morning, take the dish out of the fridge while you preheat the oven to 350°F. Let the casserole sit at room temperature for 20 minutes to remove the chill.
Add a crunchy top: If you like a crisp finish, sprinkle turbinado sugar evenly over the surface.
Bake: Cover with foil and bake for 25 minutes. Remove the foil and bake another 20 to 25 minutes, until the top is golden and the center is set with only a slight jiggle.
An instant-read thermometer in the center should read about 175°F to 185°F.
Rest and serve: Let it rest for 10 minutes. Dust with powdered sugar, slice into squares, and serve with warm maple syrup.