Preheat your oven to 400°F (200°C). Line a baking sheet with a silicone baking mat suitable for broiling, or prepare the baking sheet without the mat for easier cleanup afterward.
In a large mixing bowl, whisk together the all-purpose flour, white whole wheat flour, brown sugar, baking powder, cinnamon, nutmeg, allspice, and salt. Work in the cold butter until the mixture has a crumbly texture resembling coarse crumbs.
In a separate small bowl, combine the milk, egg, and vanilla extract. Pour this mixture into the dry ingredients and stir just until incorporated. Gently fold in the chopped apple.
Using a 1/4 cup measuring scoop, drop portions of the batter onto the prepared baking sheet. For smaller fritters, use a heaping tablespoon of batter for each portion.
Bake for 11-13 minutes, or until the bottoms are golden and the centers are fully set. If making mini fritters, start checking for doneness after 8 minutes. Transfer the baked fritters to a wire rack and switch the oven to the broil setting.
While the oven is heating to broil, prepare the glaze. Whisk together the confectioner’s sugar, milk, and vanilla extract until smooth. Lightly brush the fritters with a thin layer of the glaze and place them under the broiler for 2-4 minutes until the tops turn a light golden color and the glaze starts to bubble.
Remove the baking sheet from the oven and brush the fritters with the remaining glaze while still warm. Let the glaze harden slightly before serving.