With Facebook’s “Resilient Signals,” advertising surveillance evolves

How Facebook circumvents your protections against tracking, with the complicity of advertisers

Published by Pixel de Tracking on August 3, 2021

Advertising, the Facebook product

Facebook's financial results are crystal clear, more than 98% of its revenue comes from advertising. Facebook has 2 growth levers here:

  • Increase supply: namely, providing more advertising opportunities to advertisers.
  • Increase demand: namely, convincing existing advertisers to increase their advertising budgets, and signing new advertisers.

Increase supply: your available brain time

The most direct way to increase advertising supply is to acquire new users. As this internal memo shows, Facebook pushed this strategy of growth at all costs to the extreme:

That can be bad if they make it negative. Maybe it costs a life by exposing someone to bullies. Maybe someone dies in a terrorist attack coordinated on our tools.

[...] The ugly truth is that we believe in connecting people so deeply that anything that allows us to connect more people more often is de facto good. It is perhaps the only area where the metrics do tell the true story as far as we are concerned.

Another lever is diversification through the acquisition of existing apps. In 2012, Facebook bought Instagram, its users and its know-how:

instagram

Zuckerberg on the acquisition of Instagram: Facebook buys itself time in order to incorporate Instagram's growth mechanisms into its apps (at the time, Instagram was ahead on mobile and photos).

Increasing advertising supply also means monetizing third-party inventory: Facebook offers apps the option of using its ad network, the “Facebook Audience Network”, in exchange for a commission. Note that monetizing third-party inventory depends on the ability to read the user's Facebook data, and this is probably one of the reasons that pushed Facebook to shut down its ad network on the mobile web. With Intelligent Tracking Prevention (ITP), Safari users have protections against Facebook tracking on third-party sites.

Having billions of users on Facebook or Instagram is not enough; Facebook also has to make them dependent, whatever the cost. Here is a good summary from Robin Kelly, US Senator from Illinois:

The business model for your platforms is quite simple: keep users engaged. The more time people spend on social media, the more data harvested and targeted ads sold. To build that engagement, social media platforms amplify content that gets attention. That can be cat videos or vacation pictures, but too often it means content that’s incendiary, contains conspiracy theories or violence. Algorithms on the platforms can actively funnel users from the mainstream to the fringe, subjecting users to more extreme content, all to maintain user engagement.

One last lever: always show ads (compared with posts from your friends and the pages you follow). If you are still on Instagram, you may have noticed the gradual invasion of ads in your feed and stories. Instagram has caught up with Facebook, and we have probably reached the saturation point. Even if Facebook and Instagram do not really have any competitors, ad pressure is at its maximum.

Increasing demand: consumerism and propaganda

Offering numerous advertising opportunities is not enough; advertisers still have to be convinced that ads work on Facebook apps and across its partner network. Facebook has several strengths: its ability to target a large share of the population at different times of the day, its many targeting criteria, and its "native" ad formats. But the key point is profitability: Facebook must prove that it makes money for advertisers, and not the other way around.

How does Facebook optimize an advertising campaign? Here is a standard process:

  1. Start the ad campaign without prior targeting: this allows Facebook to reach users and test advertising messages.
  2. Then observe conversions: in order to understand the characteristics of users who achieve the advertiser's objective, the most effective ad messages, the best time and context for delivering the ad, etc.
  3. Finally, target similar users: “lookalikes” or “similar audiences” in adtech jargon.

Thus, the effectiveness of Facebook's advertising campaigns is closely linked to the information it holds about its users as well as its ability to correctly measure conversions.

manager

By default, the advertiser indicates their goal, budget and message; Facebook's algorithm handles targeting automatically.

Quite often, Facebook knows in advance which type of users to target because the advertiser has sent it a customer list called a "custom audience" and asked it to target similar users.

hearing

The advertiser can let Facebook find the right audiences, upload a “custom audience”, or manually enter targeting criteria. They instantly see an estimate of the number of people reached.

Put like that, it may sound almost innocent, but Facebook targeting contributed to Trump's 2016 victory. Facebook's algorithm does not merely optimize ad campaigns, it also promotes the spread of large-scale propaganda and therefore represents a serious threat to democracies.

Facebook is spying on you, everywhere

On its apps

The general public knows the Facebook app. But Facebook also owns Instagram and WhatsApp, among the most widely used apps in the world. Facebook also has a foot in the future with Oculus, its virtual reality platform, where it would also very much like to show ads. Each of your interactions is used to make you dependent and monetize your attention.

At this point, you can delete your Facebook and Instagram accounts, move your contacts from WhatsApp to Signal, and not buy an Oculus headset. One trap with Facebook and Instagram: you should not simply deactivate your accounts, but delete them.

Everywhere else

What the general public does not know is that Facebook also spies on you elsewhere. The list of tools made available to third parties is endless, and they come with a trade-off: letting Facebook collect ever more information about you, for its own use.

Here are some examples, not exhaustive:

Most sites and apps use at least one of Facebook's tools. And since you are rarely asked for your opinion, Facebook often knows what you do, even if you do not have a Facebook or Instagram account. Apple's video ad for App Tracking Transparency is a good illustration of Facebook's invasive side.

Since early 2020 and in response to multiple scandals, Facebook has allowed users to see their "off-Facebook" activity and disconnect it from their accounts:

offsite

Don't be fooled by Facebook propaganda, "sometimes" means "usually."

Disconnecting the activity does not mean deleting it from Facebook's servers... The company keeps all your interactions, but no longer associates them with your account. Note that Facebook will not make it easy for you: to disconnect your activity, you will need to:

It would be so nice if it were a little more visible and done in just one step... Like on your Google account, for example:

history

Google, the other Big Brother, makes your life a little easier.

Obviously, neither Facebook (nor Google) deletes your activity, it is too valuable. It's simply a matter of no longer associating it with your account.

What if you do not have a Facebook or Instagram account? Facebook still tracks you, has your contact details via the address books it may have siphoned from your friends, and carefully stores your activity on the web, in apps and offline. What is the value for its advertising model?

  • Facebook measures the impact of its ads by comparing conversion rates between people exposed to the ad and people not exposed.
  • You may be exposed to Facebook Audience Network ads on third-party apps.
  • More generally, your behavior allows the Facebook algorithm to improve its predictive models.

Surveillance in danger

As Internet users become increasingly aware of privacy protection, surveillance outside Facebook apps is becoming more difficult. We will look at how Facebook adapts and communicates with advertisers through the white paper "Why you should leverage Facebook's resilient signals", co-written with the French agency Fifty-Five.

Why communicate with advertisers? Facebook needs accomplices to track you outside its apps; advertisers must correctly use the tools made available by Facebook so that your personal data flows back properly and durably (the famous "resilient signals") to the Menlo Park giant.

First of all, why would Facebook's surveillance outside of its apps be in danger? For 3 reasons according to Facebook and Fifty-Five:

crumble

What impact for advertisers' advertising campaigns?

impact

Facebook's current trackers, pixels and other SDKs, are not very resilient signals. We also see the opportunity for Facebook to extend its surveillance to what you do offline.

As we can see:

  • Facebook can continue to measure what you do on its apps (Facebook, Instagram).
  • Facebook struggles to measure what you do on advertisers' sites and apps: due to browser protections and adblockers, old Facebook trackers are less and less effective.
  • Facebook only hasa very partial view of what you do offline.

How is this partial view problematic for advertisers? Here, Facebook and Fifty-Five need to explain why widespread surveillance is supposedly necessary:

precise

Here, Facebook tries to reassure you with very generic targeting criteria. In reality, your advertising profile is incredibly more detailed.

Measuring advertising performance is essential. If Facebook has access to fewer conversions (purchases, registrations, installations, etc.), its algorithm will have less information about what works (user profiles, ad messages, advertising context), which will reduce the effectiveness of its ads. The advertiser may end up paying more for worse results.

How do you scare advertisers? Tell them that if they do not allow Facebook to track you well enough, their ad campaigns will no longer work properly:

feat

“Holes” in the measurement among advertisers? A disaster according to Facebook...

If the advertiser does not fill these measurement “holes”, here are the consequences:

  • Fewer conversions attributed to Facebook: some conversions will not be measured, and reporting will show ad campaigns as less effective than they actually are, with a higher customer acquisition cost.
  • A truly less effective advertising campaign: Facebook will have less information to optimize the campaign.
  • Imperfect information: as a result, the advertiser will doubt the reporting for potential future ad campaigns.

“Resilient signals”, or how Facebook circumvents your protections

How does Facebook allow advertisers to fill these “holes”? Through what it calls “resilient signals”:

resilient

With your adblocker, did you think you were protected from Facebook's invasive surveillance? Wrong...

With the complicity of advertisers, here is how Facebook circumvents your protections:

When Apple forces its hand, Facebook ads become more respectful of your privacy

The Aggregated Event Measurement tool (also called AEM): allows Facebook to measure the effectiveness of ad campaigns in an aggregated way (256 ad campaigns maximum, no individual tracking) when the user has refused Facebook or Instagram tracking on iOS (via ATT). It is inspired by Apple's solution, WebKit Private Click Measurement or PCM, and it is the only privacy-friendly solution offered by Facebook.

You might be surprised, knowing Facebook: why not track iOS users on the web when they have refused tracking in the Facebook or Instagram apps? Because in its guidelines for app developers, Apple is very clear:

Tracking refers to the act of linking user or device data collected from your app with user or device data collected from other companies’ apps, websites, or offline properties for targeted advertising or advertising measurement purposes

If Facebook violated this rule, it would risk being kicked out of the App Store. You can delve deeper into the subject with the article “Understanding Facebook’s updated iOS14 advertising guidance”.

And why not use PCM directly? Mystery, Facebook only declares:

Our solution is similar to Apple's private click measurement tool. However, it addresses some key use cases by advertisers not covered by Apple.

Note that if you refuse Facebook or Instagram tracking via the iOS ATT window, Facebook says it reflects this choice when processing events sent via the Conversions API:

Events sent to Facebook via the Conversions API will also be processed in accordance with the limits defined by the Aggregated Event Measurement tool.

How does Facebook handle events from other “resilient signals” (advanced matching and offline conversions)? Mystery...

Note that Facebook can also make proposals when it comes to advertising standards that better respect privacy. Some discussions between Apple and Facebook engineers (namely John Wilander and Ben Savage) sometimes make it possible to move beyond the Apple vs. Facebook conflict, with proposals that address advertising use cases while preserving Internet users' privacy:

Browser developments intended to better protect privacy are discussed within the W3C's "Privacy Community Group", the organization responsible for co-building web standards. Facebook is therefore playing its role here: trying to influence browser developments in order to limit the impact on its advertising business. And for that, Facebook needs credibility with engineers from Apple, Firefox or Brave. Their first principle is respect for Internet users' privacy (Chrome engineers often have Google's advertising bias).

On the subject of the W3C and this war for influence, read the excellent article "Concern trolls and power grabs: Inside Big Tech’s angry, geeky, often petty war for your privacy".

Illustrated circumventions, examples of advertising campaigns

To illustrate the impact of “resilient signals”, Facebook and Fifty-Five give 2 examples.

User sees an ad but doesn't click

First case: the user sees an ad on Facebook, does not click on it, but then goes to the advertiser's site to complete a form. They will be called back by a call center, and eventually register.

example1

Since this user's browser (Safari or Firefox, for example) blocks Facebook's third-party cookies, the Facebook tracker (pixel) cannot read the Facebook user ID on the advertiser's website. Facebook is therefore unable to link the user's ad exposure with their browsing on the advertiser's site.

But when the user fills out the form, the advertiser can call the Conversions API or use advanced matching for the web. The advertiser can then send Facebook the personal identification data from the form (name, email address, phone number, etc.) and Facebook can link it to the user's account. Likewise, after the user registers by phone, the advertiser will use the Conversions API to transmit the registration to Facebook.

User clicks on an ad

Second case: the user is on Safari, clicks on a Facebook ad, then visits one of the advertiser's product pages 8 days later before finally buying offline.

example2

When the user clicks on the Facebook ad, Facebook manages to place a first-party cookie on the advertiser's site. The Facebook pixel installed on the advertiser's site retrieves the click ID fbclid contained in the URL parameter, and stores it in the first-party cookie _fbc. Note that Safari could perfectly well delete these tracking parameters, as Brave has already been doing since last year.

Since 2019 and the ITP 2.1 update, Safari deletes cookies created client-side after 7 days (here, the _fbc cookie created by the Facebook pixel). When the user visits the advertiser's product page after 8 days, Facebook can no longer link it to the initial click; it does not recognize the user.

Facebook and Fifty-Five point to the workaround: the prerequisite is to use server-side tagging, such as Google Tag Manager's server-side setup, which lets you create an HTTP cookie via the HTTP Set-Cookie header and thus bypass the 7-day lifespan of client-side cookies.

The next step is to configure the Conversions API to work with Google Tag Manager, an action already very well documented by Facebook and by Simo Ahava.

So when the user visits the product page, Google Tag Manager is called with a persistent first-party cookie, the mapping with the click identifier fbclid is correctly recorded, and the Google Tag Manager server container can correctly call Facebook's Conversions API to pass on the information. Additional “benefit”: bypassing adblockers! Then, when the user buys in-store, the advertiser uses the offline conversions tool to send the user's purchases to Facebook.

Surveillance cheat sheets

To push advertisers to adopt its “resilient signals”, Facebook took advantage of Fifty-Five's remarkable work. The white paper therefore contains cheat sheets for each “signal”.

ecosystem

A summary of the different “resilient signals”.

The Conversions API

Here is the cheat sheet for the marketing team:

capi

Facebook and Fifty-Five recommend installing the Conversions API alongside the Facebook pixel, and therefore collecting events both client-side and server-side.

The Conversions API thus makes it possible to bypass protections in browsers such as Safari (the 7-day limit on cookies created in JavaScript). It also helps bypass adblockers when used together with server-side tagging. Finally, this API makes it possible to transmit offline information, such as user scoring.

Here is the cheat sheet for the technical team:

capi2

One piece of advice stands out in particular: "CAPI is a way to secure data for sensitive sectors (Banking, Insurance, ...)". And yes, your banking or insurance data leaks to Facebook, but don't worry, the connection is secure! Once again, everything is done to facilitate the leaking of all your interactions, particularly via native integrations:

  • Tag Managers for your interactions with the advertiser's site
  • eCommerce, CRM and Customer Data Platforms for your offline purchases and interactions.

Highlighting a native integration with Google Tag Manager:

server

The integration of the Conversions API with Google Tag Manager's Server-Side Tagging is promoted by Facebook, as Simo Ahava points out.

Advanced matching

Here is the cheat sheet for the marketing team:

matching1

This option makes it possible to bypass browsers' blocking of third-party cookies and leak your personal data even if you have never clicked on a Facebook ad. In practice, the advertiser sends your personal identification data to Facebook, which allows Facebook to link it to your Facebook account.

In the limitations, we can note:

  • "Although cookie-less, it is key you collect consent from users": Do Facebook and Fifty-Five consider that your consent is not necessary with the other "resilient signals"?
  • "If you work in sensitive sectors such as Banking, a heavier setup is required": you must then use manual advanced matching rather than automatic matching, meaning you configure the pixel (IMG) yourself to send personal identification data instead of letting the Facebook pixel (JavaScript) automatically retrieve the right fields from the form.

Here is the cheat sheet for the technical team:

matching2

Nothing new, except the reminder that manual advanced matching must be used for "sensitive" sectors.

Offline conversions

Here is the cheat sheet for the marketing team:

offline1

Nothing new here either, only the observation that surveillance continues offline, relying on your personal identification data (name, email, phone number, etc.).

Here is the cheat sheet for the technical team:

offline2

Once again, we can note that advertisers can import offline conversions themselves, or go through one of Facebook's many partners. Facebook's surveillance capitalism works thanks to a vast ecosystem.

The Aggregated Event Measurement (AEM) tool

Here is the cheat sheet for the marketing team:

aem

This setting allows the advertiser and Facebook to “limit the damage” for iOS users who refuse tracking. Facebook reports aggregated performance information, which allows it to optimize ad campaigns.

Note that Facebook can very well learn from campaigns running on Android and from iOS users who accept tracking, in order to better understand what works and what does not (user profiles and ad messages), and thus optimize campaigns for iOS users who refuse tracking.

Facebook and Google, the precursors of adtech

As we have seen, Facebook has implemented a series of tools to bypass your protections and track you better: “resilient signals”. These tools are offered to advertisers turnkey, with complete documentation:

Facebook's surveillance complex is impressive, but Google is no slouch. Facebook's Conversions API can rely on Google Tag Manager Server-Side Tagging infrastructure to bypass adblockers and browser protections. And Google is evolving its own surveillance based on the Conversions API model with its "enhanced conversions".

These new practices will unfortunately be a source of inspiration for other advertising players, making advertising surveillance even more ubiquitous and difficult to avoid.

What can we do? This arms race (surveillance marketing vs. browsers and adblockers) is not enough; it will never protect the least informed. We therefore need to ban targeted advertising directly in law: