A focus on portrait mode: behind the scenes with Pixel 2’s camera features

This week the Pixel 2 and Pixel 2 XL, Google’s newest smartphones, arrive in stores. Both devices come with features like Now Playing, the Google Assistant, and the best-rated smartphone camera ever, according to DXO.

We designed Pixel 2’s camera by asking how we can make the camera in your Pixel 2 act like SLRs and other big cameras, and we’ve talked before about the tech we use to do that (such as HDR+). Today we’re highlighting a new feature for Pixel 2’s camera: portrait mode.

With portrait mode, you can take pictures of your friends and family (that includes your pets too!) that keep what’s important sharp and in focus, but softly blur out the background. This helps draw your attention to the people in your pictures and keeps you from getting distracted by what’s in the background. This works on both Pixel 2 and Pixel 2 XL, on both the rear- and front-facing cameras.

Technically, blurring out the background like this is an effect called “shallow depth of field.” The big lenses on SLRs can be configured to do this by changing their aperture, but smartphone cameras have fixed, small apertures that produce images where everything is more or less sharp. To create this effect with a smartphone camera, we need to know which parts of the image are far away in order to blur them artificially.

Normally, to determine what’s far away with a smartphone camera you’d need to use two cameras close to each other, then triangulate the depth of various parts of the scene—just like your eyes work. But on Pixel 2 we’re able to combine computational photography and machine learning to do the same with just one camera.

How portrait mode works on the Pixel 2

Portrait mode starts with an HDR+ picture where everything is sharp and high-quality.

Next, our technology needs to decide which pixels belong to the foreground of the image (a person, or your dog) and which belong to the background. This is called a “segmentation mask” and it’s where machine learning comes in. We trained a neural network to look at a picture and understand which pixels are people and which aren’t. Because photos of people may also include things like hats, sunglasses, and ice cream cones, we trained our network on close to a million pictures—including pictures with things like those!

Just creating two layers—foreground and background, with a hard edge in between them—isn’t quite enough for all pictures you’d want to take; SLRs produce blur that gets stronger with each fraction of an inch further from the thing that’s in sharp focus. To recreate that look with Pixel 2’s rear camera, we use the new Dual Pixel sensor to look through the left and right sides of the camera’s tiny lens at the same time—effectively giving us two cameras for the price of one. Using these two views we compute a depth map: the distance from the camera to each point in the scene. Then we blur the image based on the combination of the depth map and the segmentation mask.

The result? Portrait mode.

Portrait mode works a little differently on the front-facing camera, where we aren’t able to produce a depth map the same way we do with the more powerful rear-facing camera. For selfies, we just use our segmentation mask, which works particularly well for selfies since they have simpler compositions.

When and how to use portrait mode

Portrait mode on the Pixel 2 is automatic and easy to use—just choose it from your camera menu then take your picture. You can use it for pictures of your friends, family, and even pets. You can also use it for all kinds of “close-up” shots of objects such flowers, food, or bumblebees (just don’t get stung!) with background blur.

Here are some tips for how to take great portraits using any camera (and Pixel 2 as well!):

• Stand close enough to your subjects that their head (or head and shoulders) fill the frame.
  
• For a group shot where you want everyone sharp, place them at the same distance from the camera.
  
• Put some distance between your subjects and the background.
  
• For close-up shots, tap to focus to get more control over what’s sharp and what’s blurred. Also, the camera can’t focus on things closer than several inches, so stay at least that far away.

_{Best smartphone camera claim is accurate as of the date this post was published, and is subject to change.}