This is an area which is not covered much (or not at all) by text books or articles. Why? I think it's mainly because it's harder to test 3D graphics than testing numeric algorithms or database manipulation. Another reason is that the web application community seems to have been better at picking up TDD than the scientific or game development community.
It's not impossible, though. So where do we start? Let's have a look on a scenario where we want to develop an application with a 3D scatter plot of a 4 dimensional dataset. The plot has the following requirements:
- All data samples shall be represented as spheres in 3D space
- The first 3 dimensions shall be defined by the spatial X/Y/Z position in the plot
- The 4th dimension shall be indicated with a color
- In order to ble able to focus on a specific area and minimize the cluttering of the display, the user shall be able to interactively move a box-shaped probe in the plot and make the points outside of this box smaller.
These are typical requirements for a scientific application, but the principles for testing it can be applied to geological 3D models, medical data, games or other kinds of 3D graphics.
 |
| The plot should look like this (left). The user is focusing on a smaller area (right). |
Know what you are testing
Testing 3D graphics can seem a bit daunting. How do you verify that the graphics card is producing the correct pixels animated on the screen? The short answer is: usually you shouldn't.
Keep in mind the pillars of good unit tests: test the right thing. Also keep in mind the Single Responsibility Principle. What is the visualization code under test doing? Is it actually producing pixels, or is it using a 3D rendering toolkit to do the visualization?
Keep in mind the pillars of good unit tests: test the right thing. Also keep in mind the Single Responsibility Principle. What is the visualization code under test doing? Is it actually producing pixels, or is it using a 3D rendering toolkit to do the visualization?
High-level visualization
3D visualization software often use a high-level 3D toolkit like Open Inventor, VTK or HueSpace to do the 3D rendering. In this case, you should trust that the 3D toolkit is rendering correctly whatever you instruct it to render. Your code is creating a scenegraph or a visual decision tree, and the 3D toolkit is doing the rendering based on this.Let's say that we have a dataset and data sample class that looks like this:
public struct DataSample
{
public double ValueDim1;
public double ValueDim2;
public double ValueDim3;
public double ValueDim4;
}
public class Dataset
{
public event ChangedEventHandler DataSamplesChanged;
public IEnumerable<DataSample> DataSamples { get; private set; }
...and the rest of the implementation here
}The plot view is a class which takes a dataset as constructor parameter and produces an Open Inventor scenegraph. A naive Open Inventor implementation might look like this:
public class ScatterPlotView
{
private Dataset _dataset;
public SoSeparator OivNode { get; private set; }
public ScatterPlotView(Dataset dataset)
{
_dataset = dataset;
this.OivNode = new SoSeparator();
UpdateNode();
}
private void UpdateNode()
{
this.OivNode.RemoveAllChildren();
foreach (var dataSample in _dataset.DataSamples)
{
var sampleRoot = new SoSeparator();
var color = new SoMaterial();
sampleRoot.AddChild(color);
color.diffuseColor.SetValue(GetColorByValue(dataSample.ValueDim4));
var translation = new SoTranslation();
sampleRoot.AddChild(translation);
translation.translation.Value = new SbVec3f(dataSample.ValueDim1, dataSample.ValueDim2, dataSample.ValueDim3);
var sphere = new SoSphere();
sphere.radius.Value = GetRadiusBasedOnWhetherSampleIsInsideProbe(...);
sampleRoot.AddChild(sphere);
this.OivNode.AddChild(sampleRoot);
}
}
private SbVec3f GetColorByValue(double valueDim4)
{
// Look up color in a color table
}
}There are many scenarios that we might want to test here, but let's have a look on one specific scenario: the datapoint changes, and the scenegraph should change accordingly.
[Test]
public void OivNode_DatasetChanges_SceneGraphIsUpdated()
{
// Arrange
var dataset = new Dataset();
dataset.DataSamples = new[]
{
new DataSample(),
new DataSample()
}; // Initial samples
var scatterPlotView = new ScatterPlotView(dataset);
// Act
dataset.DataSamples = new[]
{
new DataSample(),
new DataSample(),
new DataSample()
}; // Set 3 other samples
// Assert
Assert.AreEqual(3, scatterPlotView.OivNode.GetNumChildren());
} Here, we create a plot view with a dataset that has two samples. The dataset is then modified to have three samples, and the test verifies that the scenegraph changes accordingly. Note that we don't inspect the scenegraph in detail here. This test verifies that the scene graph is modified when the dataset changes and should assert on only that.
Other tests might traverse the scene graph and verify the position and color of each of the spheres in the scene graph. Just make sure that you don't overspecify the test. Don't write a test that will fail if the color scale changes slightly! In general you should test that the scene graph behaves correctly, rather than re-creating the logic in the scene graph construction.
User interaction testing
So far we have tested that the plot reacts to changes in the data. How about user interaction testing? This is actually similar to the previous test: make an action and assess the scene graph. The difference is how the action is performed: we need to mimic user interaction.
Again - know what you are testing! If you are using Open Inventor draggers, you don't need to emulate the mouse. That is not your responsibility -- it's Open Inventor's responsibility to transform mouse movements into dragger movements!
Let's write a test that verifies that the probe behaves correctly. The probe is represented with a SoTabBoxDragger which is added to the scene graph by the ScatterPlotView class. Here is one example of a test:
[Test]
public void OivNode_ProbeIsDragged_DataPointsOutsideBoxAreSmaller()
{
// Arrange
var dataset = new Dataset();
dataset.DataSamples = new[]
{
new DataSample { ValueDim1 = 0.1 },
new DataSample { ValueDim2 = 0.9 }
};
var scatterPlotView = new ScatterPlotView(dataset);
// Act
scatterPlotView.BoxDragger.translation.Value = new SbVec3f(0.5f, 0, 0);
// Assert
var sphereForSample1 = ...traverse the scene graph to find first sphere
var sphereForSample2 = ...traverse the scene graph to find second sphere
Assert.AreEqual(0.1, sphereForSample1.radius.Value);
Assert.AreEqual(1.0, sphereForSample2.radius.Value);
}Here we insert two points into the dataset. The dragger is moved so that in only contains one of the points, and the test inspects the scene graph to verify that the sphere sizes are correct.
If you write your own draggers instead of using Open Inventor's built-in draggers, you may need to emulate actual mouse coordinates. Still, you should make abstractions so that you can pass synthetic mouse events to the nodes rather than emulating actual mouse events.
Scene graph inspection
How do we verify that the scene graph is correct? One might create a very rigid test that verifies every node and node connection. That quickly leads to an overspecified test which reproduces the logic in the production code, however.
For Open Inventor, it's practical to use a SoSearchAction and inspect the resulting path(s). VTK has a similar mechanism for inspecting the pipeline. Just make sure that you don't copy the logic of the production code.
The scene graph related to the data samples in our plotter can be inspected like this:
var searchAction = new SoSearchAction();
searchAction.SetInterest(SoSearchAction.Interests.ALL);
searchAction.SetType(typeof(SoSphere));
searchAction.SetSearchingAll(true);
searchAction.Apply(scatterPlotView.OivNode);
// Get all the paths that lead to a SoSphere:
var pathList = searchAction.GetPaths();
int pathCount = pathList.Count;
// We can use the path count to assert against the number of samples
foreach (SoPath path in pathList)
{
var sphereRoot = path.GetNode(path.Length-2) as SoSeparator;
bool hasTranslation = false;
for (int i = 0; i < sphereRoot.GetNumChildren(); ++i)
{
if (sphereRoot.GetChild(i) is SoTranslation)
{
// We can assert that the sphere is positioned with
// a translation. We can also check the actual position
hasTranslation = true;
}
}
}In this example, we inspect the relevant part of the scene graph and verify that
- There are N paths leading to data point spheres, which should equal to N samples
- The sphere positions are determined by a SoTranslation. We could also check the actual position of the SoTranslation
This could be extended to verify that the correct material is assigned to the relevant spheres. The possibilities are endless, but consider splitting up the test so that each test is asserting on only one responsibility.
Inspection code like this makes a test hard to read, and should be put into helper methods. A test call would look like this:
Inspection code like this makes a test hard to read, and should be put into helper methods. A test call would look like this:
var pathsLeadingToSphere = OivTestHelper.GetPaths(plotView.OivRoot, typeof(SoSphere));
Assert.AreEqual(3, pathsLeadingToSphere.Count);
Assert.IsTrue(OivTestHelper.PathContainsNode(pathsLeadingToSphere[0], typeof(SoTranslation));Of course, you will usually not traverse the entire scene graph of a view at once. In complex views, you should divide the scene graph into smaller parts that can be tested individually.
Low-level visualization
For low-level visualization like ray tracing or fragment shaders where your code is actually responsible for doing the rendering, a test-first first approach is a bit harder. It's difficult to write a test that specifies what the result should be because you are producing a bitmap rather than a state.The first step would be to ensure that you have a good separation between the low-level rendering code and the higher-level tier that is setting up states and handling user interaction, as the latter can be unit tested more easily.
I haven't found any practical approach to do proper TDD of low level visualization, but you can write regression tests that verify that the results are still correct after optimizations, generalizations and bug fixes. Given that the rendering code is able to render to a bitmap, you can do bitmap comparisons and compare future test sessions to a set of reference images.
Summary
Like I mentioned at the beginning of this post, unit testing 3D graphics is not trivial. The most important point is that you should test the state or the scene graph of the rendering engine and trust that the rendering engine does its job translating this into pixels. If you don't trust it, perhaps it's a good idea to use something else...
One might argue that this scene graph traversal and inspection introduces logic and complexity to the tests. This contradicts somewhat with what I have written in Pillars of Unit Tests. I try to mitigate this by creating helper methods for scene graph traversal. There is some logic involved, but this logic is hidden and doesn't obscure the readability of the tests. When you think about it, this is conceptually the same as calling NUnit's equality comparison for arrays or other non-trivial equality checks.
No comments:
Post a Comment