Sony A9 focus accuracy – birds in flight hit rate

In an earlier post I compared the Sony A9 focus accuracy with that of the Olympus EM1x to get a comparative birds in flight hit rate. It has caused a certain amount of incredulity or outright dismissal on various forums around the world, particularly those oriented towards Sony cameras. Like Trump supporters and the US elections, some enthusiasts cannot accept that a) the Olympus can perform better than the Sony, and b) that the Sony only achieved a hit rate accuracy of 49% in my tests. Clearly my tests must be wrong (although others have found the same problems). Only for ultra nerds, but let’s look at the tests and the data in detail for the A9 alone.

Quick look at the problem

Before going into detail, here is a 2 minute video that shows the problem. This is a fast run-through of the focal point positions of several hundred images. All the Exif data is shown on the right. This is using the latest v2.6 of the amazing (and misleadingly named) A7info.exe program. When I did the original analysis of the data below, in August 2020, A7info did not work too well with the A9, so I had to use the much slower and less elegant AFV.exe program, which can take up to 10 seconds per image, and recording the focus accuracy of over 1500 images led to me losing the will to live. The latest version of A7info is quite fantastic, and you can fly through images at Photomechanic speeds. As a result I was able to take a screen video capture of all the shots. You can stop the movie at any point to see the EXIF and focus info for any shot. You will see that there are a huge number of misses, and even where there are focus hits, the bird is often out of focus.

AF Stress tests

I did these tests at the Hawk Conservancy Trust (HCT) in Hampshire, UK. The HTC birds are  vultures, falcons, kestrels, owls, eagles and kites. There is a huge range of shooting conditions, from large birds flying fast right at you (and sometimes hitting the camera) to aerial raptor combat, or super fast falcon  dives at very long range.  A Lanner falcon is about the size of wrapped cheese sandwich and flies at over 100mph, and it’s a challenge getting even a focus point on one, so some of these tests are quite extreme.

The HCT provides a pretty good location for testing autofocus hit rate.  There are two displays daily, using more or less the same birds flying in the same sequence in the same way. This is about as close as you can get to a demanding but repeatable test setup for birds in-flight.

There is a morning session in which birds fly fast towards or across the camera, quite close in, using focal lengths between 150 mm and 450 mm. The starting position of the birds is almost always known.

The afternoon session has extremely fast and unpredictable bird movements where either the speeds are extreme as with the falcons, or the movement is incredibly sudden and you have maybe 1-5 seconds to focus, compose, and shoot with the kites.  The climax of the kite flying is a feeding frenzy where chicken pieces are catapulted up in the air, past flying kites, and they fly upwards to chase them, and jostle each other out of the way in the process. They take maybe 1 second to find the chicken piece, and can remain static in the air for a millisecond, before swooping down.

The goal is to catch them in the air at that split second, or as they chase each other down as in this album. By the way, I have never seen any other photos of this stage of the kite feeding anywhere on the internet, although I am sure they must exist. The afternoon focal lengths are usually around 600 to 800 mm, so there is the double problem of first finding the bird with a very narrow field of view, and then following it. It is incredibly hard to do.

As a rough generalisation, the morning session tests the ability to lock focus over a sequence of shots, and the afternoon tests the ability to instantly acquire focus and capture very sudden movement.  The latter is a very tough test but quite representative of places like RSPB Bempton cliffs for example, where you have literally only seconds to grab shots of extremely fast moving Puffins or Gannets.

Camera setup

I tested the A9 with firmware 6.0, using the Sony 200-600 f5.6-6.3 lens. I imported Mark Smith’s excellent settings for the camera (Mark is an astonishingly good pro bird photographer and A9 fan). The AF was usually zone, with 20 fps electronic shutter.  I shot in manual, wide open, at 1/2000 with auto ISO set to 6400 max, and used rear dial exposure compensation plus highlight/shadow indication to get the exposure right for the bird. Over the course of a morning and afternoon session I took 1460 shots with the 200-600. I also took a further 160 with the 200-600 and the Sony 1.4 TC adapter for the afternoon falcon flying.

Tracking the subject

I cannot show all 1600 shots, but to give you an idea, here are screenshots of the 126 (of 456 total) thumbnails of the out-of-focus (OOF) images for the morning session.. You will note that the bird is front and centre in almost every image, and it is not operator error or technique that is causing the problems.

Here is the same approach for the afternoon session, for the 616 (of 1003 total) OOF images.

And here is the same for the 1.4 TC falcon session.

So just to make it clear, every bird is well in frame in every session, and they should all be capable of being in focus.

Checking Focus accuracy

I checked the focus accuracy of every image using FastRawViewer at 1:1 in the original raw format, with additional sharpness applied and using focus peaking tools, and exposure adjustments where necessary. I have written about my focus accuracy checking technique extensively in this post and the method was the same. I used FastRawViewer’s XMP/ Lightroom colour labels  to identify photographs as being either out of focus (red), in focus (green) or nearly in focus (yellow).  

Checking AF point location

The focus accuracy tests as we will see below were pretty disappointing for the A9. Had I just missed focus for all these? The thumbnails above say not, but to be sure, as I had done for the A7R4, I checked all 1460 shots for focus point location using the AFV.exe program, and recorded the results by coding the metadata in FastRawViewer. This is a mind numbingly tedious process, which you can’t do too often or you would go mad. AFV.exe takes the focus point location information from the RAW image EXIF data, and displays this on the embedded .JPG image. It shows the final (orange) focus point (FP), and additional (green) AF points. I will show some examples below.

I used FastRawViewer’s XMP/ Lightroom rating labels  to identify photographs as having the orange FP and/or the green AF point on the subject as rating 3. Photos where one or more green AF points were on the birds, but the final focus point was not (but bloody well should have been), were given a 4 rating. Images where there were no FPs or AF pts on the bird, or inexplicably, where there was no FP data at all, were given a 1 rating.

The filtering capability of FastRawViewer enables  subtotals to be easily shown  –  and an example of the overall results display can be seen in the screenshot image below. If you look at the table on the right of the screenshot, you can see we are looking at out of focus shots (red + yellow) where the AF data says focus lock was gained (***). There are 164 unsatisfactory shots where the focus should have been perfect. Click the image for a full screen lightbox.

The results

OK, here is the data in tabular form for the entire test.

Results summary: A9 focus accuracy – birds in flight hit rate

Here are the headline figures. The overall % of in focus shots across all 1460 for the morning and afternoon sessions combined was 49%. This was heavily influenced by the afternoon session, which was 2/3rds of the total, and for which the A9 is not well suited (I have not include the even worse results from the 1.4 TC tests in this figure, which would reduce the shots accuracy percentage even further). The number of out of focus shots in that session was 61%, a pretty dreadful result. That is probably because, despite all the birds being clearly in the frame, it only laid a focus point on them 50% of the time in the afternoon session.

Even when the focus was perfect, (with the Final Focus point and other AF points on the bird), 48% of the A9 shots were out of focus in the afternoon session. Overall, of the 815 images with an AF point on subject, which should have been in focus, 37% or 299 images were out of focus. Over the whole day, 742 shots were out of focus. I could have done better with my phone………

Examples of Sony A9 plus 200-600 f5.6-6.3 missing focus

Here are some examples of where the focus point was right on the bird but the image was out of focus. This is particularly egregious because everything was perfect and the camera totally failed to perform. On the testing day, there were 299 images where the camera failed in this way.

Now here are 50 examples of where the bird should have been in focus, but the camera laid all the the focus points in thin air. These examples of missed focus are fantastically annoying, as many of them are the ‘golden moments, with the bird captured in that split second stationary feeding pose. All f*cked by the A9 not performing well. On the testing day, there were 443 images where the camera failed in this way.

Sony A9 plus 200-600 f5.6-6.3 and 1.4 TC results and examples

As noted, I also tested the A9 with the 200-600 f5.6 plus the Sony 1.4 TC for the afternoon falcon and vulture flying, where a long focal length is required, and the Sony could in theory get out to 900mm.

The falcon test is quite demanding as the bird flies fast, and at a long distance away. The starting position is usually known though so these are slightly easier tracking shots than the kite feeding ones. The focal lengths were in practice between 500mm and 700mm FFE, the lens was wide open at F9, and the shutter speed was 1/2000, with ISOs between 2000 and 3500. The results were very poor, with 64% being out of focus.

Here is series of shots from this series, showing how the A9 completely failed to get focus.

I also took a sequence of shots of a large vulture flying slowly from right to left with the A9, and the 200-600 plus 1.4 TC. The focal length was an FF equivalent of 840mm, the lens was wide open at F9, and the shutter speed was 1/2000 s, with ISOs of 6400. In this series of 52 shots, there was only 1 in focus (no table needed really). This series shows 35 of them. At the beginning there are no green AF points activated, only the orange final focal point. The orange FP loses the bird after 3 or 4 shots. Then you see the AF points and the FP at the right hand side of the frame, miles from the bird. Then they randomly dot around the screen, almost always missing the subject. It doesn’t begin to find the subject until shot 26 of the series, and even then it keeps missing the bird. All the shots were out of focus on inspection. This is quite instructive as to how the A9 focus really fails over a prolonged series of images. The camera and lens combination may have failed because it only has simple linear phase detect AF sensors and not cross-type (as in the EM1x), or because at low light and small apertures it defaults to contrast detection as discussed in this post. Either way it was an unacceptable performance.

Even if the shots had been in focus, I could not have used the images, as the resolution and noise levels were dreadful. These results, the poor performance of the A6400, and the disastrous performance of the A7R4 mean that for all practical purposes the Sony A9 is capped at a maximum of 600 mm focal length for me. 

How can these results be so different from Mirrorless Comparisons?

There has been a lot of forum comment on the impossibility (aside from my implicit incompetence) of only getting a 50% hit rate with the A9, citing the test results from the excellent Mathieu Gasquet, who has reported 90% accuracy. I hope you can see from the above that my shooting technique and testing methodology are pretty rigorous. How could there be this disparity then?

I am familiar with Bwlch Nant Yr Arian in Wales, UK, where Mattieu takes his test shots. The birds are large kites, often in the hundreds, at a relatively constant distance of around 30-50 m. You choose your bird from many candidates, lock focus, fire your shots and voila, 90%!  It’s not a challenging situation. In similar (but tougher) circumstances at the HCT with large birds, the A9 had better focus accuracy than the E-M1x in my tests, by roughly the same margin as Mattieu obtained, but at a lower overall hit rate (71% in my tests vs 90% in his). But most of the shooting situations at the HCT and in my own shooting experience, are much more challenging than at Bwlch Nant Yr Arian. In particular in the HCT afternoon session it is extremely difficult even to follow the birds, let alone lay a focus point on them.

It was in these afternoon sessions where the EM1x was better, in situations where you have to immediately focus and shoot a bird whose position is changing wildly and quickly and you have maybe 1-3 seconds to get the picture. In this situation, which is common when a single bird is being shot in the wild, the A9 was slower to gain focus and on many occasions did not get the shot (39% accuracy cf 45% on the EM1x), as shown above.  As far as I know (and as described by Matthieu), that kind of shooting situation was not tested by Mirrorlessons. Other bird photographers have noted for the A9 and the 200-600, “for quick acquisition with changing focus distances….the Sony system is dead in the water most of the time”

I would also note that Matteiu also rates the Sony A6400 very highly, and higher than the EM1x. I took over 3400 shots over 7 sessions with the A6400 and found it had far worse focus accuracy than the EM1x, as I show in this post. So, although Mirrorless comparisons is a fabulous resource, Mattieu’s conclusions absolutely did not apply for me on any of the Sony cameras I tested (the A7R4, A9, A6400 and A7R3), and under the challenging conditions I shoot in.

What other reason could there be for these results?

Some have said that I had a bad copy of my A9. Possibly, but then my A9 and my A7R4, and my A6400 were all bad copies. I guess I am willing think this of Sony, but I have never had a bad copy of an Olympus, Nikon, Fuji or Canon camera, and I have owned a very large number of them.

Possibly a bad copy of the lens? But this was my second copy of the 200-600, as I had returned the first one after the debacle of the A7R4. Also, Mirrorlessons (ML) got a 90% hit rate with the A9, as discussed above. I got a 72% hit rate with the A9 at the Hawk Conservancy Trust, Andover (HCT) in the morning session. In both situations there was plenty of time to get an initial focus lock, which favours the A9. The difference between my 72% and ML’s 90% is almost certainly because in the HCT session, birds are also flying directly at you at close range, which does not happen at the kite feeding station. As my HCT morning session test results with the A9 and 200-600 lens are consistent with the results of Mirrorlessons, it seems clear that the lens was not defective.

Or maybe, these are representative results for the Sony cameras with the 200-600 lens. No-one else (as far as I know) has published  a rigorous analysis like this in such challenging circumstances, and actually checked the focus accuracy and focus point location on nearly 1500 images.   I would love to see someone do a similar challenging test, and when the weather improves, I will repeat mine.  But I would suggest politely to incredulous readers and A9 fans that just because you don’t like the results, that doesn’t mean they are incorrect.