Put all the fruit in a large bowl, along with the grated orange zest. Pour over the sherry and mix well. Cover and leave overnight.
Preheat the oven to 150°C/gas mark 2. Line an 8-inch cake tin with parchment paper. Put the butter and sugar in a large bowl and beat together with an electric whisk until light and creamy. Beat in the eggs one at a time, adding a little flour between each addition to prevent curdling. Carefully fold in the remaining flour, baking powder and spices until well combined, then fold in the ground almonds. Fold through the marinated fruit.
Spoon the cake into the prepared cake tin and level the surface. Bake in the oven for around 2 hours, or until a cake skewer comes out clean. Remove from the oven and leave to cool in the tin for 10 minutes. Remove from the tin and leave to cool completely. Wrap in foil until ready to ice.
Put the butter, brandy and double cream in a freestanding mixer and beat until combined. Slowly add the icing sugar, beating well between each addition. Spread the buttercream all over the cake with a palette knife.
Wash the fruit and pat it dry with a paper towel. Lightly whisk the egg white in a bowl, then brush the fruit with egg white using a pastry brush. Spread the caster sugar on a plate, then roll the fruit in the sugar. Leave to set on a wire rack. Decorate the cake with the frosted fruit.