We describe a technique to simultaneously estimate a local neural fiber model and trace out its path. Existing techniques estimate the local fiber orientation at each voxel independently so there is no running knowledge of confidence in the estimated fiber model. We formulate fiber tracking as recursive estimation: at each step of tracing the fiber, the current estimate is guided by the previous. To do this we model the signal as a mixture of Gaussian tensors and perform tractography within a filter framework. Starting from a seed point, each fiber is traced to its termination using an unscented Kalman filter to simultaneously fit the local model and propagate in the most consistent direction. Despite the presence of noise and uncertainty, this provides a causal estimate of the local structure at each point along the fiber. Synthetic experiments demonstrate that this approach reduces signal reconstruction error and significantly improves the angular resolution at crossings and branchings. In vivo experiments confirm the ability to trace out fibers in areas known to contain such crossing and branching while providing inherent path regularization.