There isn't an easy way to do this: the group style is defined in the grid's control template and you need to retemplate the grid to override it. If you're already retemplating the grid then it should be reasonably easy to change your existing GroupStyle to start out collapsed rather than expanded. Otherwise you'll need to create a whole new control template.
If you need to do this, the basic grid template looks like this:
<Border Background="{TemplateBinding Background}"
BorderBrush="{TemplateBinding BorderBrush}"
BorderThickness="{TemplateBinding BorderThickness}">
<DockPanel>
<StackPanel DockPanel.Dock="Top"
Visibility="{Binding IsToolBarVisible, RelativeSource={RelativeSource AncestorType={x:Type ms:PropertyGrid}}, Converter={StaticResource BooleanToVisibility}}">
<ms:ViewCommandsToolBar Target="{Binding BindingView, RelativeSource={RelativeSource AncestorType={x:Type ms:PropertyGrid}}}" />
<TextBlock HorizontalAlignment="Center"
TextWrapping="Wrap"
Padding="0,20,0,20"
Visibility="{Binding BindingView.DefaultView.IsEmpty, RelativeSource={RelativeSource AncestorType={x:Type ms:PropertyGrid}}, Converter={StaticResource BooleanToVisibility}}">
No properties match the search criteria.
</TextBlock>
</StackPanel>
<ms:TreeListView x:Name="PART_Grid"
ItemTemplate="{StaticResource PropertyGridHierarchy}"
ItemContainerStyle="{StaticResource {x:Static ms:PropertyGrid.SelectionTrackingStyleKey}}"
ItemsSource="{TemplateBinding BindingView}">
<ms:TreeListView.Columns>
<GridViewColumnCollection>
<GridViewColumn CellTemplate="{Binding PropertyNameTemplate, RelativeSource={RelativeSource AncestorType={x:Type ms:PropertyGrid}}}" />
<GridViewColumn CellTemplateSelector="{Binding RelativeSource={RelativeSource AncestorType={x:Type ms:PropertyGrid}}, Path=EditorSelector}" />
</GridViewColumnCollection>
</ms:TreeListView.Columns>
</ms:TreeListView>
</DockPanel>
</Border>
You will also need the following resources:
<BooleanToVisibilityConverter x:Key="BooleanToVisibility" />
<HierarchicalDataTemplate x:Key="PropertyGridHierarchy" ItemsSource="{Binding Path=Children}" />
This probably looks a bit intimidating, but half of it is just setting up the toolbar. The grid itself is just the TreeListView which is reasonably simple -- ask if you have any questions.
Now for the group style. Add the following property element to the TreeListView declaration:
<local:TreeListView.GroupStyle>
<GroupStyle>
<GroupStyle.ContainerStyle>
<Style TargetType="{x:Type GroupItem}">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type GroupItem}">
<Expander Style="{StaticResource TreeListViewStyleExpander}" IsExpanded="True">
<Expander.Header>
<TextBlock Background="{DynamicResource {x:Static SystemColors.ControlLightBrushKey}}"
Text="{Binding Name}" FontWeight="Bold" Margin="2" />
</Expander.Header>
<Expander.Content>
<ItemsPresenter />
</Expander.Content>
</Expander>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
</GroupStyle.ContainerStyle>
</GroupStyle>
</local:TreeListView.GroupStyle>
Again, this looks lengthy but most of it is XAML noise. All the work is being done by the Expander. In our default style, as you can see, the Expander's IsExpanded starts out True. You just need to change this to False.
Finally, if you want to keep the default appearance of the expand/collapse button, you'll also need the following style resource:
<Style x:Key="TreeListViewStyleExpander" TargetType="{x:Type Expander}">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type Expander}">
<StackPanel>
<DockPanel Background="{DynamicResource {x:Static SystemColors.ControlBrushKey}}">
<ToggleButton x:Name="Expander" DockPanel.Dock="Left"
Style="{StaticResource {x:Static local:TreeListView.ExpandCollapseToggleStyleKey}}"
IsChecked="{Binding IsExpanded, Mode=TwoWay, RelativeSource={RelativeSource TemplatedParent}}"
ClickMode="Press" Margin="-2,0,-2,0" />
<ContentPresenter ContentSource="Header" />
</DockPanel>
<ContentPresenter Visibility="{TemplateBinding IsExpanded, Converter={StaticResource BooleanToVisibility}}"/>
</StackPanel>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
I realise this is a bit of an epic! We will take a look at providing extensibility points to do this stuff a bit more easily but for the time being I'm afraid this is the way to go about it.