iPhone 15 Pro Lab Testing: Rolling Shutter, Dynamic Range, and Exposure Latitude Analysis

Upon release of the new iPhone 15 Pro and iPhone 15 Pro Max there was quite a hype around the filmmaking capabilities of this new phone. Looking at the specs and codecs we felt quite intrigued to get our hands on this new iPhone and run it through our lab torture test and test the iPhone 15 Pro’s dynamic range, rolling shutter and exposure latitude. How will it stack up compared to the recent breed of dedicated hybrid consumer or even professional cinema cameras? Well, let’s find out… In life there is always a first and that also applies to camera lab testing – so this is the first time we are testing a phone in the lab. So far it wouldn’t have made much sense, as typically a lot of image manipulation is happening in phones in an automated way, such as auto tone mapping which tones down bright parts of the image and increases brightness in the shadows. With the introduction of a log mode, namely Apple LOG for the latest iPhone 15 Pro & Max this situation has changed. With LOG, there is now the required consistency for a rigorous test. So, let’s jump right into it – or not? Well, it’s not so easy to be honest, because we always try to stick to some very rigid principles in our testing. So first, at that point again a very big “thank you” for the nice collaboration with my colleague Florian Milz. Especially when testing the front facing camera things got a bit complicated but he will always find a way … Testing the front facing camera in our CineD latitude test. Image credit: CineD Challenge accepted: adjusting CineD Lab Test standard to iPhone 15 Pro Coming back to our rigid standards, for one, we always try to use the same Zeiss 50mm CP2 T/2.1 macro lens for APS-C cameras, or the Zeiss 85mm T1.5 lens for full frame cameras. Using these focal lengths ensures a certain distance from the Xyla 21 stepchart (or subject in the latitude test) to avoid internal reflections from the chart which might impact lower stops in the dynamic range evaluation. Please have a look at our article here explaining how we measure dynamic range. Secondly we always do a proper research to find out native ISO’s of sensors, and then obviously dial in the white balance, shutter, ISO and focus plane manually to ensure consistency. Now to item one, on the iPhone 15 Pro there are 13mm, 24mm and 77mm equiv. focal lengths available on the three different back facing cameras. On the iPhone 15 Pro Max there is a 120mm focal length instead of the 77mm. Strangely, for the front facing camera there is no focal length mentioned. So you will get all results from us below, with the exception of the 13mm camera. We found just too much reflection from the XYLA 21 chart using this camera. Blackmagic Camera App to the rescue To the second point, the native Apple camera app does not provide any possibility to set manual parameters. So we had to resort to the Blackmagic Camera app which allows exactly that. Settings in the app for all the lab tests were Apple Log and 4K ProRes HQ. The Blackmagic App taps into the hardware image pipeline hence it receives the same information that the native Apple camera app has. Also, we had to find out what the native ISO on each of the four camera’s sensors (including the front facing one) is. Ahead of our Lab Test, we were able to determine that the lowest ISO (which is different on each of the 4 cameras) is always the cleanest (native) ISO. Higher ISO’s would just push up the code values thus resulting in brighter images, but also noisy shadows. However, higher ISO’s also bring up the difference between lower stops in Apple Log, so there is a potential benefit of shooting at higher ISO’s if separation in darker stops is needed. Perfect. Time to get going, right? Rolling Shutter of the iPhone 15 Pro / Max Cameras Let’s start with the iPhone 15 Pro 24mm camera. Using our 300Hz strobe light we get 5.3ms (less is better) of rolling shutter: 24mm camera rolling shutter on the iPhone 15 Pro. Image credit: CineD The 13mm camera reads 4.7ms, the 77mm camera 5ms and the 120mm camera (Max) again 5ms. The front facing camera reads 9.3ms to our surprise: Front facing camera rolling shutter on the iPhone 15 Pro. Image credit: CineD Comparing these full sensor read out values to recent consumer or even professional cinema cameras, the ~5ms is a superb result. There is only one camera in the market that tops it: the Sony VENICE 2 with less than 3ms. Alternatively, there are global shutter camera options that have 0ms like the RED KOMODO (and most likely, the newly announced Sony a9 III full frame camera which comes with a global shutter – but we have yet to get our hands on this camera to verify this). Dynamic Range of the iPhone 15 Pro / Max Let me quickly repeat again why we have 3 ways to judge the dynamic range of a camera: The waveform plot of the Xyla 21 chart at native sensor resolution on a timeline with that resolution: this gives a visual indication of how many stops can be identified above the noise floor (= the usable stops). Also, it shows the code value distribution of the stops. Very often, the lower stops are very close together in terms of code value (the Y – axis), hence if you underexpose and raise the shadows later in post (i.e. expanding the shadow stops), you will not have enough code values between the stops and the result is ugly banding (loss of fine color transitions between the stops). IMATEST: IMATEST will calculate the signal to noise ratio for each stop. That is a purely mathematical calculation and comes in handy to identify, how “clean” each of the stops is. Cameras that use a lot of internal noise reduction naturally fare better than others with less noise reduction. There is no way to account for that in a meaningful way, as the noise footprint of every camera / sensor is different. Hence, there is also no “standard” noise reduction that you can apply in post to compare cameras. That’s why we at CineD always turn off the noise reduction – as recommended by IMATEST. Latitude: exposure latitude is the capability of a camera to retain colors and detail when over- or underexposed. Our CineD studio scene is the real world test (in a controlled environment) of how far a camera can be pushed. The beauty of this test is that it clearly reveals how many stops are usable in our carefully composed standard scene, as the playfield gets equalized – and, it is revealed if a camera is “cheating” by using excessive internal noise reduction. No matter how much internal / or post noise reduction is used, cameras that show a solid 12 stops at a signal-to-noise ratio of 2 typically have 8 stops of latitude in our scene. Cameras like the ARRI ALEXA that show 2 stops more at SNR = 2 also have two stops more latitude. Cameras that try to achieve something close to 12 stops at SNR = 2 by using heavy internal noise reduction only show 6 – 7 stops of latitude. So, this trinity of tests is very revealing and helps to identify if the combination of sensor readout, signal processing, and codec allows to push exposure around over a large range. iPhone 15 Pro 24mm camera at ISO55 – dynamic range The waveform shows around 11 stops above the noise floor. Speaking of which, there is almost no such thing as a noise floor – everything is super clean, hinting at massive noise reduction happening internally (there is no way to turn this “off”): 24mm Waveform of the XYLA 21 chart at ISO55. Image credit: CineD Wow, 11 stops are really good. Now let’s run this through IMATEST: IMATEST result for the 24mm camera at ISO55. Image credit: CineD We are getting 12 stops of dynamic range in the iPhone 15 Pro (Max) for a signal to noise ratio (SNR) of 1, and the same 12 stops for a signal to noise ratio of 2. Also for the “slope based DR”. This is a clear sign of “too much” noise processing for IMATEST to calculate a meaningful result. It also becomes obvious in the lowest diagram where “Noise (% of max pixel)” is shown. Noise values for the shadow stops are super low. Now let’s increase ISO to see if we get some differentiated result with IMATEST. Looking at ISO1200, we get the following waveform for the 24mm camera: 24mm Waveform of the XYLA 21 chart at ISO1200. Image credit: Cine Comparing this waveform at ISO1200 to the waveform at ISO55 you can see clearly how the code values (Y-axis) are shifted upwards to increase brightness in the image. Now also the darker stops are more differentiated and you can see a 12th stop emerging. It will be interesting to see if the latitude test will reflect this difference as well. IMATEST calculates (higher) 13.4 stops at SNR = 2 and 13.4 stops at SNR = 1. IMATEST result for the 24mm camera at ISO55. Image credit: CineD Just looking at these results could lead you to conclude that the iPhone 15 Pro reaches ARRI Alexa levels of dynamic range (have a look at our ARRI ALEXA classic and Mini LF test here and our ALEXA 35 test here). Well… My conclusion so far is that IMATEST mainly measures the noise reduction, and not so much the “real” dynamic range. We will have clarity once we move to the latitude section. Spoiler alert: at the end of the day it is a cell phone with tiny sensors … Here is a table of IMATEST results for the other cameras of iPhone 15 Pro / Pro Max: IMATEST dynamic range results for the other iPhone 15 Pro / Max cameras. Image credit: CineD Exposure Latitude results of the iPhone 15 Pro Max As described earlier, latitude is the capability of a camera to retain colors and details when over / underexposed and pushed back to a base reference level. For our CineD studio tests, the base exposure level refers to a luma waveform value of around 60% on the forehead of my dear colleague Johnnie. We always establish the clipping level first by overexposing until the red channel is at the cusp of clipping on the forehead of our subject. This means that some colors have already clipped on the colorchecker on the left. From there we then underexpose in 1 stop increments. We did this via the Blackmagic app using the shutter value (1/30s, 1/60s, etc…). As mentioned earlier everything was shot in Apple Log and 4K ProRes HQ on the iPhone Pro Max phone (at the time of exposure testing we only had this one available). For the iPhone 15 tests we used the 24mm and the front-facing camera, and we checked exposure levels in the RGB waveform. The beauty is that an external monitor can be connected as a visual reference: Using an external monitor to check RGB waveform levels. Image credit: CineD Now, to develop the shots we used DaVinci Resolve 18.6, via a color space transform (CST, from REC2020 and Apple Log to REC709). Adjustments to exposure or noise reduction were always made on the first node, the second node then did the CST: Color space transformation of the REC 2020 Apple Log to Rec709. Image credit: CineD 24mm at ISO55 – our base exposure looks like that: Base exposure of our standard CineD studio scene with 24mm. Image credit: CineD Now, at ISO55 can go to 3 stops over and bring back the image to base exposure in post (using the lift / gamma / gain primaries in DaVinci Resolve 18.6) without any issues: 3 stops over and pushed back, 24mm ISO55. Image credit: CineD Now, moving to two stops of underexposure and bringing back the image in post, we get already a rather noisy image: 2 stops under and pushed back, 24mm ISO55. Image credit: CineD This is already at the cusp of being usable. Let’s push it one stop more: 3 stops under and pushed back, 24mm ISO55. Image credit: CineD Uhh … that doesn’t look good. We have reached the latitude limit. Massive banding can be seen, as well as a very blotchy chroma noise distribution. Image sharpness is lost as well. Also, the concentric circles hint at an in – camera vignetting compensation. If our suspicion, derived from the IMATEST results is right we will not be able to do much with noise reduction, as the image is already heavily noise processed by default. Hence, let’s apply noise reduction to the 2 stops under and 3 stops under images: 2 stops under and pushed back using noise reduction, 24mm ISO55. Image credit: CineD 3 stops under and pushed back using noise reduction, 24mm ISO55. Image credit: CineD Please have a look further down in the article for a table of the noise reduction settings applied in DaVinci Resolve 18.6 for the various ISO’s and cameras. Concluding for the 24mm camera at ISO55, we get 5 stops of exposure latitude (3 above to 2 under). This is actually 2 if not 3 stops below the current crop of consumer APS-C or full-frame cameras. Comparing it to the aforementioned ARRI Alexa Mini LF, it is 5 stops less exposure latitude. And compared to the Alexa 35, the difference is even seven stops. Now let’s have a look at higher ISO’s for the 24mm camera – at ISO1200 As mentioned earlier in the dynamic range section, we noticed the phenomenon that at higher ISO’s code values would shift up and the lower (darker) stops would show a bit more differentiation resulting in potentially less banding. Let’s directly move to the 3 stops under image, for ISO1208 and 24mm camera: 3 stops under and pushed back, 24mm ISO1208. Image credit: CineD 3 stops under and pushed back using noise reduction, 24mm ISO1208. Image credit: Cine Well, this actually looks a bit better than at ISO55. Still, I would not consider this usable. Now let’s look at the front-facing camera starting with ISO55 We couldn’t use the lowest of ISO 20 as it was not possible to clip the forehead of Johnnie with our standard studio lighting. Again, 3 stops overexposure from base is easily possible, so let’s move directly to 2 stops under: 2 stops under and pushed back, Front Facing at ISO55. Image credit: CineD Interestingly, the front-facing camera has a different, more organic image processing, also the noise is a bit finer, not so blotchy (especially the chroma noise) as with the 24mm camera. This image cleans up nicely again: 2 stops under and pushed back using noise reduction, front-facing camera at ISO55. Image credit: CineD Let’s see if we can push this to 3 stops under: 3 stops under and pushed back, front-facing camera at ISO55. Image credit: CineD This looks better actually than the 24mm at 3 stops under, even at ISO1200. Now, let’s do one more test at 3 stops under, using ISO1207 on the front-facing camera: 3 stops under and pushed back, front-facing camera at ISO1207. Image credit: CineD It becomes very obvious that the noise is much finer, especially the chroma noise. However, there is much more luma noise in the image. So let’s apply noise reduction: 3 stops under and pushed back using NR, front-facing camera at ISO1207. Image credit: CineD Ok, this is still not usable unfortunately. There is less of the blotchy chroma noise and much more luma noise, but overall there is a pinkish tint and the image is still not usable. Noise reduction settings in DaVinci Resolve. Image credit: CineD As a summary, we can conclude that the iPhone 15 Pro / Max is capable of 5 stops of exposure latitude, with some wiggle room towards 6 at higher ISO’s. Summary Quite a funny experience to run a phone through our lab tests! The rolling shutter is exceptionally good on the iPhone 15 Pro / Max, with around 5ms for all cameras, the front-facing camera has 9.3ms which is still very good. Perfect for handheld video shooting. Looking at the waveforms and the iPhone 15 Pro dynamic range in IMATEST, at first glance you could be tricked into believing the very high values that are displayed. In the end, it turns out that those high IMATEST values are achieved with a really high internal noise reduction, confirmed by the latitude results of 5 stops (which is on the very low end in our benchmark). Recent Micro four-thirds cameras like the GH6 have seven stops of exposure latitude, consumer APS-C cameras like the FUJIFILM X-H2S or the Sony A6700 have 8 stops and the leader of the pack, the ARRI Alexa 35 has 12 stops to give you the benchmark of our testing. Nevertheless, we have to keep things in perspective – we are talking about a phone with tiny image sensors … So all due respect to Apple in bringing Apple Log with 4k ProRes HQ encoding to their latest phones, which opens up a lot of possibilities for creators. It will only get better from here. Exciting. Have you shot video on the iPhone 15 Pro / Max? What are your experiences? Please let us know in the comments section below.