Diffusion Tensor Magnetic Resonance Imaging (DTI) enables the exploration of fibrous tissues such as brain white matter and muscles non-invasively *in-*vivo [7]. It exploits the fact that water in these tissues diffuses at faster rates along fibers than orthogonal to them. Integral curves that estimate fiber tracts by showing paths of fastest diffusion are among the most common information derived from DTI volumes. The ability to estimate fiber tracts *in-*vivo is one of the key advantages of DTI over other imaging techniques. Integral curves are generated from DTI data by following the principal eigenvector of the underlying diffusion tensor field bidirectionally. These curves are often visualized with streamlines or variations of streamlines (streamtubes and hyper-streamlines) in 3D [28], [42]. Reflecting the intricacy of the connectivity in the brain, these 3D models are generally visually dense. Therefore, it is often dif ficult to ascertain tract projections as well as anatomical and functional structures clearly. Consequently, typical interaction tasks over tracts, such as fine bundle selection, are often dif ficult to perform and have been a focus of recent research [1], [2].

In this context, we present a method for visualizing and interacting with fiber tracts alongside abstract, lower-dimensional representations.

For a set of tracts, given as lists of connected points, we create conventional 3D streamtube models and two 2D representations: a planar embedding and a hierarchical clustering tree. Both 2D visualizations are representations of a similarity (af finity) matrix obtained by computing pairwise "distances" between the fiber tracts. We obtain the planar embedding by considering each fiber tract to be an individual 2D point for which we compute coordinates that approximately reflect the distance relations between the fiber tracts as represented in the similarity matrix. We compute the tree representation, or dendogram, by applying the average linkage hierarchical clustering algorithm on the similarity matrix. We link the views of these three representations implicitly through interaction and explicitly through a perceptually uniform coloring Figure 2 shows a visualization of fiber tracts obtained from a DTI brain data set in our framework. We have obtained feedback from experts in an anecdotal study for a prototype of the framework. Initial results suggest that this type of coordinated interaction has the potential to enable faster and more accurate interaction with dense fiber tract collections.

**Contributions** We introduce and anecdotally evaluate a visual exploration method that combines traditional 3D streamtube models with 2D representations, facilitating exploration of fiber tracts in the brain.

Here we discuss existing techniques related to our work. Speci fically, we present techniques for visualizing and interacting with DTI datasets, methods for visualizing similarity relations and previous work using multiple, coordinated views for visualization.

### 2.1 Visualizing and Interacting with DTI datasets

The most commonly used technique to visualize DTI data is streamline tracing; in DTI-speci fic literature this is also called fiber tracking [28] or tractography [6]. We use this method for our DTI visualization.

Interacting with streamline DTI models is not trivial. A common interaction task is the selection of fiber bundles. This is usually done directly on the model by placing 3D regions of interest (ROIs) along the presumed path of the desired bundle and then having the application select fibers that intersect those ROIs [12], [40], [26]. More recently, Akers et al. [2] introduced a 3D sketching and gesture interface for pathway selection: the user paints a 2D freehand stroke and the selection algorithm matches it to the tract bundle that looks most similar from the user's perspective. Finally, a concurrently published research by Chen et al. [14] also links 2D embeddings to DTI datasets and finds that it accelerates interaction. Our work differs mainly by incorporating hierarchical clustering trees for both exploration and embedding re finement, representing the variation data point with perceptual variation of color, and discussing several solutions to the issue of anatomical landmark. Also, our non-linear embedding process enables to embed large number of fiber tracts, including fiber tracts from whole brain data sets.

Automatic DTI fiber clustering methods have been developed to support DTI model interaction and visualization. For a review of such methods consult [27]. The measure used to de fine the geometric similarity between integral curves is fundamental for clustering. A number of such measures have been described and can be divided into two categories: those that measure the Euclidean distance between two selected points on two curves and those that summarize all points along two curves as the mean Euclidean distance along their arc lengths.

Examples of the former type of proximity measure include the closest point measure, the Hausdorff distance [16], and the Fréchet distance [3]. Examples of the latter type include the average point-by-point distance between corresponding segments de fined in [18], the mean of closest distances de fined in [16], and the mean of thresholded closest distances de fined in [42]. Brun et al. [9] embed the curves in a feature space and then calculate the distance between the two curves in the feature space. Finally, fiber similarity can be mapped to color as was first done in [10] by assigning distinct colors to clusters, and more recently in [17] by immersing a 3D embedding into the L*a*b* color space.

### 2.2 Visualizing Similarity

The visualization literature describes several methods for conveying similarity relationships between entities. Most of them have been researched in the context of multidimensional visualization, where the distance is derived from the position of a point along each dimension. However, a subset of these methods can be used for entities over which an arbitrary similarity function is speci fied. In the following, we will only review this category. For a more detailed discussion on multi-dimensional visualization techniques, Keim [25] provides a good overview.

An intuitive way of making distance apparent is by using a scatterplot, one of the types of visualization we use in this work. In its simplest form however, this method can only be used for data with at most three dimensions and explicit vector values. To overcome this limitation, multi-dimensional scaling (MDS) techniques have been developed. They attempt to map the multi-dimensional points to a visualizable lower dimension while preserving distance relations between points. Methods in this category fall into two categories: linear and non-linear. Linear methods perform a linear combination of the original high-dimensional values to project them to a new basis of smaller dimension. A well known such example is Principal Component Analysis (PCA) [23].

Non-linear methods are suited for computing representations when distances between points are given explicitly but coordinate values for the points are unknown, as is the case in the tract similarities computed as part of this research. These methods use the distance between data points to de fine an error measure that quanti fies the amount of distance information lost during the embedding. Gradient descent or force simulation is then used to arrange the points in the low dimensional space so as to minimize the error measure. A good example of such an approach is Force Directed Placement (FDP) [20] originally proposed by Eades [19] as a graph drawing approach. It simulates a system of masses connected by springs of lengths equal to the distances that need to be embedded. The points are initially placed at random and are then iteratively moved by displacements derived from forces computed by Hook's spring laws. After a number of iterations the spring system will reach a local minimum energy state that represents the resulting embedding. We use this method as part of our work.

An iteration of the original FDP model is *O*(*n*^{2}), and since at least *n* iterations are necessary to reach equilibrium, the final complexity is *O*(*n*^{3}). This makes the computation for high-resolution, complete brain models expensive. One method that addresses this problem is called Force Scheme. Proposed by Tejada et al. [38], it reduces the overall complexity to *O*(*n*^{2}) by requiring fewer iterations to reach the final state. A complexity of *O*(*n*^{5}/4) was achieved by Morrison et al. [29] by creating a hybrid model based on approximations using samples and interpolations. In this paper we use another algorithm, with linear iteration time, developed by Chalmers [13].

A MDS can be used in conjunction with a perceptually uniform color space to display similarity as a color cue. We use this technique to reflect the variation of tract similarity as a perceptual variation of colors: similar tracts receive perceptually similar colors while dissimilar tracts get perceptually distant colors. A color space is said to be perceptually uniform if the perceptual difference between any two colors in just noticeable difference (JND) units is equal to the Euclidean distance between the two colors in that color space. The L*a*b* color space is perceptually uniform and thus a 2D or 3D embedding can be immersed into L*a*b* to obtain a similarity color coding. It should be noted, however, that the perceptually uniformity in the L*a*b* is an empirical approximation and assumes a particular calibration setting for individual monitors.

A dendogram is another method for visualizing similarity that does not require explicit vector values for points and as such is suited for displaying tract similarity. It is a tree-like visual representation of results produced by hierarchical agglomerative clustering algorithms such as [4] or [24]. Because they are used in a wide range of scienti fic domains they have become intuitive tools for many scientists.

### 2.3 Coordinated Views for Visualization

Visualization techniques are usually task and data speci fic. Different views are therefore frequently used to show data from multiple perspectives, combine the strengths of any individual technique, and distribute the cognitive and interpretative load of complicated data and tasks across multiple views [5].

However, the task of aggregating the different views into a unitary single mental image factors in the complexity of the visualization itself [5]. This effect can be reduced by coordinating the content, appearance and behavior of the views [30]. This is achieved either implicitly, through coordinated appearance or behavior, or explicitly through visual cues, such as color or lines linking the separate windows. In this paper we use both approaches. Shneiderman [37] offers a good review on multiple-view coordination techniques such as brushing and linking or details on demand.

As in our work, multiple views applications are often used to aid in the understanding and exploration of complicated datasets. In [11], the authors show several examples of how brushing and linking techniques can be used to map a complicated data space into multiple simple views that, when explored together, convey the overview data picture. Gresh et al. [22] present an approach that links 3D visualizations to statistical representations to facilitate the effective exploration of medical data. XmdvTool [41] and Visulab [36] attempt to maximize a user's understanding of multidimensional data by linking multiple representational techniques such as scatterplots, glyphs, or parallel coordinates.

Finally, work such as [32] and [8] propose domain independent, extensible multiple-view architectures that satisfy general requirements of the visualization domain.

In this section we provide a detailed description of the methods we introduce and their implementation. We begin by de fining the tract similarity measure, continue with the algorithms and visualizations used to display this measure, the types of interaction that our framework permits, and conclude with details about or DTI models.

### 3.1 Similarity Between Fiber Tracts

We quantify how fiber tracts relate to each other by computing an anatomically motivated pairwise distance measure between them. Our measure tries to capture how much any given two tracts follow a similar path, while giving more weight to the points closer to tract ends. There have been different distance measures proposed for fiber tracts generated from DTI volumes [27]. In the current work, we use the weighted normalized sum of minimum distances measure proposed in [17]. We have chosen the proposed measure because it assigns higher weights to the points closer to the ends of the curves, which is important in the context of neural fiber tracts. Note that this measure does not necessarily satisfy the triangle inequality, therefore, it is not a metric. Given two integral curves *C*_{i} = {*C*_{i}^{1},...,*C*_{i}^{m} } and *C*_{j} = {*C*_{j}^{1},...,*C*_{j}^{n} } that are represented as polylines with *m* and *n* vertices respectively, we first find mean weighted distances_{i} d_{ij} and *d*_{ji}, and then determine the maximum of these two distances as the distance D_{ij} between the two curves:
TeX Source
$$d_{ij} = {1 \over m}\sum\limits_{k = 1}^m {\alpha _i^k {\rm{dist}}(C_i^k,C_j)}$$
TeX Source
$$d_{ji} = {1 \over n}\sum\limits_{k = 1}^n {\alpha _i^k {\rm{dist}}(C_j^k,C_i)}$$
TeX Source
$$D_{ij} = D_{ji} = \max \;(d_{ij},\;d_{ji})$$

The function dist(*p* *C*) returns the shortest Euclidean distance between the point *p* and curve *C*. Also , where the normalizing factor , We set the parameter σ automatically, proportional to *L*_{C} the length of the fiber tract, such that σ = λ *L*_{C}, where λ ∊ (0, 1]. We use *λ* = 0.5 for the demonstrations in this paper.

We compute distance between each pair of integral curves as we denoted and assemble the measures to create a distance matrix. While we believe our distance measure is a good approximation of the notion of similarity in the domain, the distance visualizations described in the following sections are independent of a particular distance measure. In fact, in the prototype application we enable the computation of several other distance measures.

### 3.2 Displaying Tract Similarity

We use the following visualization methods to display tract similarity: 2D embeddings, dendograms resulting from hierarchical clustering, and color coding through color embedding. In the following we discuss the particularities of each method and the interactions they enable.

#### 3.2.1 2D Embeddings

We obtain 2D embeddings of tract similarity by using Eade's [19] force directed method. To reduce the complexity of the computation and achieve interactive performance Chalmer's [13] acceleration technique is employed. Instead of computing forces on a point against every other one, two representative and disjoint sets of constant size are sampled and used for each point. At each iteration a set of *K* random points is resampled from the entire dataset and a neighbors set of maximum *M* elements is grown over all iterations by moving the closest points from the random set. Forces on a point are computed in relation with the *M* + *K* members of these limited sets, leading to a single iteration complexity of *O*(*n*). We use *M* = 10 and *K* = 20 and note that our embeddings showed little variation if these values changed.

For the force computation we use Hook's law *F* = − *k* δ *X* where δ *X* is the spring displacement and *k* is the spring constant. We experimented with variations of this force to obtain embeddings that are better suited for neurotract interaction and analysis: a sharper definition of clusters can improve bundle selection and manifold recognition while small distances should take embedding priority over large distances. We tried the following approaches: using squared distance to exaggerate large distances and make clusters more defined, using relative displacement instead of absolute difference in distance to give larger distances more arrangement flexibility, and using a combination of weighting forces with a factor inversely proportional to distance and adding a repulsive force between points. As Figure 3 shows, good visual results were obtained by combining relative distance displacement, forces weighted by a decay factor (*e*^{– σ /
d} with *σ* a decay factor and *d* the distance), and a repulsive force (*F*_{rep} = *k*_{rep}/
*d*^{2}
*embed*, where *k*_{rep} is a constant and *d*_{embed} is the embedded distance) between all points.

For a more rigorous validation we have plotted the absolute embedding error stacked on top of tube distance for an ordered subset of the smallest distances Figure 4 shows how using a decay function to weight large distances less leads to smaller error for embeddings of small distances.

Our 2D embeddings allow the following interactions: point selection, point collapsing, and coloring. Selection is performed by clicking and dragging; multiple selection can be performed to select points from non-adjacent regions. Collapsing groups a set of points into a single clustered representation. This can be used either for easier tract bundle selection or as a mechanism for manually refining embeddings − points belonging to the same tract bundle can be grouped together if the embedding algorithm places them apart. The centroid of the grouping will be used in subsequent embedding iterations. Finally, the 2D coordinates of the embedding can be interpreted as the (*a* *b*) coordinates in the L*a*b* color space, and, for a given luminance, colors can be attributed to points. The result is that close points will receive perceptually close colors. However, this color embedding is not ideal due to the particularities of the L*a*b* color gamut; it has an irregular shape and saturated colors close to the boundaries. The 2D coordinates need to be scaled to fit into the gamut and will thus occupy within the gamut a small, central region that corresponds to unsaturated colors. A result is shown in Figure 5-a.

#### 3.2.2 The 3D Color Embedder

A better coloring can be obtained, as seen in Figure 5-b, by using a 3D color embedder. We compute an approximation of the L*a*b* color gamut, as visible on the right panels of Figure 6, and use it as a container for force directed embedding. To avoid having to adjust a repulsive container force, which would likely need a hard-to-control, steep gradient, we perform a physically accurate simulation with container contact detection. The embedding begins in the center of the gamut and is gradually expanded until most of the space is filled. During implementation we observed that the largest distances are often embedded along the luminance axis (*y*-axis of color gamut). This is problematic because luminance offers little resolution and can be interpreted as a lighting effect. We therefore apply a "flattening" force at the beginning of a simulation cycle to force large distances to lie in the horizontal plane. These force components, acting on the *y*-axis towards the center of the gamut, wear off as the embedding moves towards a steady state. The force computation used is the same as for the 2D embedding, with straightforward 3D modifications. In terms of interaction, the color embedder only supports collapsing and color grabbing.

#### 3.2.3 Dendograms

Dendograms are visual representations of hierarchical trees obtained through agglomerative clustering. We use an average linkage clustering whereby the distance between two clusters is computed as the average of all inter-cluster distances. We use a few acceleration techniques to reduce the computation time: caching distances and caching candidate clusters for merging.

To compute the tree layout we use the method described in [34]: for each subtree the layouts for the two child trees are computed recursively and placed next to each other aligned at the bottom; the root is then placed one unit above their bounding box and in the middle of its horizontal axis. For single node trees a unit bounding box is used.

The following interactions are implemented for dendograms: multiple node selections, collapsing and expanding of individual nodes, or collapsing nodes automatically through cluster cuts. The top row of Figure 6 depicts a dendogram that was cut so that all clusters with tightness higher than a certain threshold were collapsed.

### 3.3 Multiple-View Interaction

We allow the user to link the types of visualization described in the previous sections so that operations performed in one view are mirrored in other views. For example, selecting points in the 2D embedding will result in a selection in the brain model, while color grabbing in the 3D color embedder will cause tracts to receive the corresponding coloring information.

Depending on its implementation, a view can either act on an operation that was passed to it or ignore it. For instance, collapsing was not implemented in the brain model view, and we have not investigated the opportunity of collapsing many 3D tracts into another representation of the entire bundle. An applicable method is described in [39] and would involve grouping collapsed tracts into fixed bundles to be used for subsequent operations on the 3D models.

Interesting interactions are possible through this architecture. Selection mirroring would probably be the most common operation, while color mirroring across all visualizations creates a perceptual correspondence that can help the users with creating mental mappings between views.

A more interesting example is that of applying the dendogram's cluster-cutting operation to collapse points in the 2D or 3D colorer Figure 6 shows the effects that a dendogram cluster cut had on a 2D embedding and 3D colorer: bundle selection is easier in the 2D view, while the embedding speed in the 3D color embedder was greatly accelerated. Collapsing points in the 3D colorer will result in a discretization of colors between fiber bundles. This might in fact be a desired effect: bundles still have perceptually close colors if they manifest similarity, but color transitions will be more abrupt and increase bundle saliency. Both the 2D embedding and the 3D colorer clusterings can be further refined interactively after a cluster cut, by expanding or collapsing individual clusters.

### 3.4 DTI Model

To generate the model used in our studies, a diffusion-weighted MRI volume of a normal volunteer's brain was acquired on a 1.5T Siemens Symphony scanner with the following acquisition parameters in 12 bipolar diffusion encoding gradient directions: thickness = 1.7mm, FOV = 21.7cm x 21.7cm, TR = 7200ms, TE = 156ms, b = 1000, and NEX = 3. The DTI volume was then obtained by fitting 6 independent parameters of a single second-order tensor at each voxel to the 12 measurements from diffusion-weighted MRI volume [7].

We have implemented a prototype of our method in C++ using G3D and Qt libraries [21], [33] and made it available online at *http://graphics.cs.brown.edu/research/sciviz/braininteraction/* Figures 1 and 2 display snapshots from our prototype.

We evaluated our methods anecdotally. We showed our prototype to a group of experts, including one research neuropsychiatrist and three neuropsychologists. They were all interested in the relationship between fiber tracts and cognitive and behavioral function in the brain. All of them have either seen or interacted with streamtube representations of fiber tracts. All have used computational tools for analyzing DTI data. Two of the experts were interested in computational tools to visualize tracts in planning gamma-knife surgery on intractable obsessive compulsive disorder (OCD) patients. The other two had research interests in vascular cognitive impairment, early Alzheimer's disease, and HIV; they focused on the corpus callosum (CC), frontal lobe, basal ganglia, cingulate bundle, superior and inferior longitudinal fasciculi, anterior internal capsule, and the uncinate fasciculus. They have been using DTI fiber tract analysis and visualization regularly for clinical research.

A think-aloud protocol was used; we demonstrated the prototype using a projector while asking questions and collecting their feedback.

There was agreement that clinical use would require additional functionality for ROI analysis and more contextual information, such as anatomical landmarks, in both the conventional view and abstract representations.

One of the experts expressed concern over learning the correspondence between the 2D point-cloud representation and the actual fiber-tract collection, arguing practitioners had limited time and inclination to learn new systems unless the tools were easily interpretable using a conventional anatomical framework.

Our experts found that the proposed paradigm can supplement the existing tools and would be particularly useful in accelerating the selection of tract bundles. They found the coloring method to be helpful and visually appealing, which was argued to be an important factor for adoption of a visualization tool. Without embedded anatomical landmarks, they found the hierarchical clustering tree to be more useful than 2D representation. One interaction scenario proposed was to select a rough region in the brain model using box selection and then gradually refine it in the hierarchical clustering tree. Also, they agreed that having transparent cut planes showing anatomical MR images while interacting with streamtube fiber models might be useful.

In this section we discuss several aspects of our work, including dimensionality reduction, evaluation, comparing different types of distance measure, and incorporating landmarks into embeddings.

### 5.1 Dimensionality Reduction

Each dimension in a visualization comes with extra cognitive and perceptual load. While there are clear advantages to three-dimensional visualizations in some contexts, previous work shows that humans are better at understanding two-dimensional representations [15], [35]. Beyond reducing cognitive and perceptual load, dimensionality reduction techniques have been popular in data mining because the "intrinsic dimensionality" of data is often much lower than the dimension of the space where data is immersed. In this context, it is not difficult to imagine fiber tracts as points on a low-dimensional manifold sitting in a high-dimensional space, particularly when we consider fiber tracts' locally continuous and smooth variation in the brain. So, we believe that low-dimensional representations go beyond being interaction gadgets and provide new "windows" into the intrinsic structure of data.

### 5.2 Evaluation

While the anecdotal study provided valuable insight into the benefits and limitations of our approach, a formal user study could help quantify improvements in terms of selection speed and accuracy.

### 5.3 Comparing Distance Measures

The 2D embeddings can also be used for a comparative analysis of multiple distance measures. In Figure 7 we show three different embeddings computed for the following distance measures: the one proposed in [17], the one used in this paper, and Haussdorf distance. We linked the embeddings to the model they were derived from. By selecting points in either embedding, changes in relationships are highlighted in the others, while the corresponding tracts are displayed in a fourth window. In Figure 7 we illustrate how a fiber bundle is embedded depending on the particular distance measure: the first type of measure uses only tract curvature and will thus place the three tracts that deviate from the bundle path further apart from the rest; the second measure adds weight to tract endpoints and as such ignores the bend in the three tracts and places all of them into the same cluster; finally, the Haussdorf distance considers only the minimum point-to-point distance and will thus place the tract-points together but also in the vicinity of other tracts that, while having close individual points, don't necessarily display any curvature similarity. We note that for complete brain models the differences in the embeddings were only local and the global aspect of the embeddings, as well as the 2D relationships between major bundles, showed very little variation.

### 5.4 Embedding Landmarks

While preserving color throughout the different visualizations helps create a mapping from 3D tracts to embedded points, the anecdotal study revealed that there is a need for including additional anatomical landmarks into the embeddings. Motivated by this, we have implemented two proof-of-concept methods for showing the ventricles in an embedding. The ventricles are connected fluid-filled cavities in the brain that are often used as landmarks.

In one method, we find the curves within an e distance to the surface of ventricles. We define the distance from a curve to the ventricle surface as the average distance from each curve-point to the closest surface point. Users can adjust the distance threshold *ε*. The curves within the *ε* neighborhood of the ventricle surface are highlighted in the 2D embedding with colored halos around tract-points and in the dendogram as vertical lines passing through tract-points and extending over the entire dendogram. In both cases, this mode of representation reflects regions of fibers that are close to the landmark surface Figure 8 shows the result of incorporating the ventricles into our lower dimensional representations using this first method. The dendogram can be extended to accept multiple landmarks by vertically stacking under or above the dendogram multiple horizontal bars delimiting regions close to landmarks. It is also important to note that it is normal for the dendogram to contain several non-contiguous regions since multiple bundles from different brain regions can be close to a single landmark.

In the second method we treat the major axis of the isosurface of the ventricles as another "tract curve" (in this case a straight line) and integrate the curve into our process Figure 9 shows the result of an embedding with using this method. One of the clear disadvantages of representing a landmark as a single point is that it does not convey the shape, orientation, or scale of the landmark in the original space. While, for simplicity, we have approximated the ventricle by its major axis more complex approximations can be thought of: a spanning tree of the ventricle mesh or the cross section of the ventricle with a midsagittal plane. Note that it would also be possible to integrate white matter landmarks (i.e., labeled clusters of fiber tracts) such as the corpus callosum, cingulum bundle, and internal capsule into our two-dimensional representations using a white matter atlas [31]. These might provide the desired context.

We presented a new method for visualizing and navigating through tractography data, combining two-dimensional representations of fiber tracts with streamtube models.

Results suggest combining traditional 3D model viewing with lower dimensional representations can ease the navigation through the complex fiber tract models, improving exploration of the connectivity in the brain. While our application interest has been in visualizing DTI fiber tracts, the paradigm presented here is general and should be applicable to other domains.

### Acknowledgments

This work was supported in part by NIH grants (1R0100415501A1, R24AI072073) and the Reisman Fellowship of the Brown Institute for Brain Sciences.