The Problem

Since the start of the OpenScope project, we aimed to turn our microscope into a fully automated, autonomously operable system. As such, it could not lack the autofocus feature. Autofocus is especially useful in certain experimental setups such as:

• time lapse imaging - to account for stage drift/settling with time

• monitoring of motile samples

• focus stacking - for imaging 3-dimensional objects

Or last, but not least, autofocus just facilitates the work of the person, operating the OpenScope. Manual focusing, although not very challenging, might still be fiddly, especially for the inexperienced user. This is when the autofocus algorithm comes to help. We tried out various methods of performing this task, and settled on the empirically best methods, although the rest of the code is present in our source code (although unused).

The Solution

The strategy of the autofocus algorithm is to calculate the focus score for each frame imaged while gradually changing the sample-objective distance, until the score is maximized. This corresponds to the most sharply focused image. The physics behind the process is very simple: image of an object, viewed through a lens, is produced at a particular distance from the lens; at any shorter/longer distance the light rays from a single point on the object do not converge into a point, and therefore the image is blurred. The focus score can therefore be simply regarded as a mathematical function with a single maximum.

There are, however, several issues to consider. Multiple methods exist for calculating the focus score. As well as this, there many ways to scan around for the best focus score. The challenge was to find the most robust solutions. Here is what we tried:

• Tested out focus scores based on variance since this was recommended by both sources [1][2] on artificially blurred out images (Gaussian blur with different standard deviations). This worked well. Variance was chosen as the focus measure to implement since it is quick to evaluate (speed being an important priority), and numpy has a function that does this already.

• Implemented an autofocus method for multiresolution search (described in the textbook [2], section 16.3.3).

  • Implemented DWT (Discrete Wavelet Transform) to decompose image to low resolution versions. This was expected to allow for faster computation and autofocusing. In practice, there was no significant computational advantage from this, so low-res images were not used in the final algorithm.

  Having been able to determine a focus score for an image, we then turned to methods for finding the most in-focus position. This involves several important issues:

  • Delays from moving the motor to receiving an updated frame - this is a common issue with feedback loops, everywhere from biological systems to adjusting the temperature of the shower. Failure to deal with this can result in oscillations around the set point, with the image remaining just out of focus most of the time. We dealt with this by introducing a delay between sending a "move motor" command and a "read focus score" command to attempt to match the two. Reading a focus score when the motors were no longer moving is not an issue, but reading a value before the desired position has been reached is. Thus, it is beneficial to err on the side of an increased counter-delay, though at the expense of the time for autofocus to take place.

  • Unreliability of motor positioning - when moving +400 steps and then -400 steps, there is no guarantee of returning to exactly the same position.

  • Local maxima not being the global maxima - A more theoretical issue than the practical ones above, it once reaching an area of high focus value, there is little guarantee that this is most focused plane in the image. This is in contrast to the issue above - if we "give up" an area of high focus, we may find it difficult to retrieve it. We ignored this issue in favour of the more practical ones above - attempting to account for global maxima of focus scores would require the autofocus procedure to be far longer than desirable. We also implemented a naive threshold to stop moving the z axis once the focus was "good enough", a value for which was empirically derived. Future work may wish to extend this to a dynamically calculated value or to characterise most-in-focus scores for many other types of samples.

  • We settled for a hill-climbing approach. Imagining the theoretical focus score for all possible z values on a graph, it may look a little like a hill. This algorithm attempts to "climb" that hill of focus scores. This algorithm assesses the focus scores of individual frames, moving in the direction where the focus is increasing. If the focus decreases, the motors are reversed in smaller increments. This process is repeated several times until an exit condition is reached.

  We tried a variety of other approaches to tackle these issues:

  • Implemented the Gaussian approximation to predict in focus position from the interval. This assumes that the image is formed from paraxial rays only, which is a good approximation in the case of microscopic imaging.

  • Implemented the Fibonacci search algorithm. It turns out that this is the optimal search algorithm for the situation of unimodal functions [2].

  • Implemented the parabola approximation to find in focus position.

Gradient search, which is a simple technique to locate local extrema, was also attempted, but did not give satisfactory results, as the learning rate required for convergence was slow. Also, noise from images drastically disrupted the calculations. This was a significant problem even when the gradient was calculated by taking multiple pictures around the point of interest.

Autofocus algorithm in action: the plot shows the increase of the variance (i.e. the focus score) of the image with each iteration.

The performance of the final autofocus algorithm varies, depending on the starting point of the search. A typical processing time is around 5s, which allows for autofocus during live-stream imaging through the Webshell.

The autofocus algorithm we have implemented is open for further improvements and performance enhancement. Ideas for future development include a dynamic threshold value to indicate the event of reaching max focus, and hence to stop the search, automatic measurement of the actual height to the CCD, better motors to improve precision etc.

References:
[1] Firestone, L., Cook, K., Culp, K., Talsania, N. and Preston, K. (1991). Comparison of autofocus methods for automated microscopy. Cytometry, 12(3), pp.195-206
[2] Wu, Q., Merchant, F. and Castleman, K. (2008). Microscope image processing. Amsterdam: Elsevier/Academic Press.