Batch upload identifiers to audience segments

You can create a file of identifiers and upload it into Google Ad Manager via the Google Cloud. Identifiers or IDs available for upload include cookies, mobile advertising IDs (AdID or IDFA), or PPIDs. Uploading IDs via the Google Cloud saves time if you need to add many of them at once.

This article covers the following topics:

• Prerequisites
• Create the upload file
• Deleting identifiers in bulk
• Upload the file
• Batch uploaded identifier expiration
• Best practices for batch uploads

Prerequisites

Before you can upload the identifiers, work with your account manager to:

• Locate and submit your network code. Find your network code in Ad Manager under Admin, then Global settings, and then Network code.
• Ensure that you've created an active first-party audience segment with which you want to associate the identifiers. If you don't want this segment to collect users using your own inventory, select "Publisher Managed" as the Population method.
  
  Expect to wait up to 24 hours from the time of segment creation to the time when you can perform a batch upload.
• Create a Google Group that contains all Google Accounts that will have access to upload and view files.

Once your account manager has completed your setup, you'll be ready to create the file of identifiers you want to upload.

Create the upload file

Required consent tag

All publishers must make updates to pass required consent signals to Ad Manager.

To confirm that consent has been gathered, add the tag process_consent in the header of any uploaded files. Rows should contain values for this header column in order to conform with CSV standards, but they will not affect processing; we recommend an empty string. Any files that are missing this phrase will be assumed to not have the proper consent from users and will fail to be processed by Google.

This update is required:

• For all users, regardless of location, as a technical requirement.
• For EU/EEA users, to adhere to the EU user consent policy and continue creating audience segments for users in the European Economic Area (EEA).

About identifiers

The batch upload process supports several different identifier types, and it's important that types of identifiers are uploaded to the segment using the correct upload file format. These identifiers fall into two categories:

• Encrypted identifiers: anything obtained from Google systems
• Raw identifiers: obtained from an external system or source

Encrypted identifiers are uploaded using the cookie_encrypted file format, while raw identifiers are uploaded in a type-specific upload file format.

Encrypted identifiers and PPIDs are network-specific. Therefore, batch upload of PPIDs or encrypted identifiers obtained from one network can't be uploaded to another network. That is, a user identifier obtained from the Data Transfer report of network A can't be batch uploaded to network B. Device identifiers, on the other hand, are not network-specific.

The following table summarizes the available identifiers and how to upload:

Using a text editor, create a plain text file and save it using any file name. You can save the file using any file extension or no extension at all.

File content requirements

• Ensure that the contents of the file are comma-separated.
• The first line must read:
  • For encrypted cookies and encrypted identifiers obtained from Data Transfer reports (Google Ad Manager 360 only): cookie_encrypted,list_id,process_consent
  • For raw Apple IDFAs: cookie_idfa,list_id,process_consent
  • For raw Android AdIDs: cookie_adid,list_id,process_consent
  • For raw PPIDs: ppid,list_id,process_consent
  • For raw Roku IDs: cookie_rida,list_id,process_consent
  • For raw tvOS IDs: cookie_tvos,list_id,process_consent
• Each subsequent line should contain an identifier, a comma, and the ID of the audience segment with which the identifier should be associated
• Each line should end with the unix line ending type linefeed (LF), NOT carriage return (CR). Depending on the method you use to produce the upload file, the line ending type is set in your text editor or in the program that produces the file.
• The encrypted identifiers are available to publishers through a Data Transfer report (Google Ad Manager 360 only). You can get the audience segment ID by signing into your Google Ad Manager account and selecting Inventory, then Audience segments.
• The "process_consent" column should be the final column in the upload file.
• Cells in the "process_consent" column may be left blank. However, it is important to include the column with the "process_consent" value set as the header.
• Batch upload files cannot be larger than 1GB in size. Files larger than this might fail to process correctly.

Example file content for encrypted cookie IDs:

cookie_encrypted,list_id,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,
w2gsrUcwxF-OiJTRmQswQA,3153490,

Example file content for raw AdIDs:

cookie_encrypted,list_id,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,
w2gsrUcwxF-OiJTRmQswQA,3153490,

Deleting identifiers in bulk

You can use the same upload file to delete identifiers in bulk by adding a column called delete. For each identifier you want to delete from the segment, enter a 1 in the delete column and for each ID you want to keep in the segment, enter a 0 in the delete column.

In the example below, the first cookie is kept in the segment, but the second cookie is disassociated from the segment with id=3153490.

cookie_encrypted,list_id,delete,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,0,
w2gsrUcwxF-OiJTRmQswQA,3153490,1,

In the following example, the first IDFA is disassociated from the segment, but the second IDFA is kept in the segment.

cookie_idfa,list_id,delete,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,1,
w2gsrUcwxF-OiJTRmQswQA,3153490,0,

Each line must contain as many comma-separated fields as are present in the header. Note that the identifiers never contain commas or other characters inconsistent with CSV format.

Keep in mind that it can take several hours for Ad Manager to process identifier uploads, whether you're adding or deleting them. When you add an identifier, you need to wait for it to be processed before you can delete it with a subsequent upload.

Upload the file

Once you've created the file, you'll need to upload it to a Google Storage bucket (subfolders are allowed). There are three ways you can access Ad Manager cloud storage buckets. In order of complexity:

• On the web: Visit https://console.developers.google.com/storage/browser/gdfp_cookieupload_[Ad Manager network code]/
• gsutil is a Python-based command-line tool that provides Unix-like commands for interacting with the storage bucket. Bucket authentication is abstracted and handled automatically.
• The Google Cloud Storage API is a full-featured API for manipulating the storage bucket, available through JSON or XML RESTful web interfaces. API client libraries are available for many popular programming environments, including Java, JavaScript, Python, and Objective-C. This approach is most useful if you need to manipulate the storage buckets programmatically to integrate with a Google App Engine app or a Java web app.

Within 24 hours, a log file will be generated in your Google Storage account with the same name as your file and a suffix indicating whether all of the identifiers in the file were processed successfully.

For example, if your file is named 1234_20130115_1, then the generated file will be named 1234_20130115_1.success or 1234_20130115_1.fail. It lists the number of identifiers processed successfully, with detailed error messages for identifiers that could not be processed. Download this log file for more detailed status information.

The results filename will appear in the following folder of your Google Storage account:

https://console.developers.google.com/storage/gdfp_cookieupload_result_[Ad Manager network code]/

After upload, the successfully processed identifiers in your file are treated as members of their associated segments when targeting inventory.

The .fail or .success file only indicates whether the file was parsed successfully; it doesn't provide meaningful information about the net increase/decrease in list size. The file is only named .success if every single identifier was uploaded successfully. However, in most cases there's at least one failure, so the output file contains the .fail suffix.

Batch uploaded identifier expiration

Identifiers uploaded as part of a Batch Upload operation obey the expiration settings that have been configured on the audience segment through the Ad Manager UI.

If the audience segment is configured with an expiration value of X days and the identifier does not satisfy the segment membership criteria within X days after upload, that identifier will expire from the audience segment after X days. Each upload operation resets the segment membership and expiration countdown for the identifiers that are uploaded.

Best practices for batch uploads

File structure

Minimize the number of files used for uploads. Note that a single file can be used to upload identifiers into multiple lists, since the list id is specified for each line in the file.

Upload throughput

We request you adhere to the following limits when uploading files:

• Upload no more than 500 files per 24 hour window
• Up to 1 GB per batch upload file
• Up to 5 GB in total batch upload files per 24 hour window

Incremental/complete uploads

• To avoid unnecessary processing, we recommend using an incremental (delta) upload of identifiers as a routine practice. You should add or delete only the identifiers whose membership in the segment has changed since the last upload, rather than uploading the full segment identifier list.
• On a less frequent basis (no more than once per week, and still subject to the upload throughput limits above), you may run a full upload of all segment identifier data. This full upload will refresh the segment membership of any members who had previously aged out of the audience segment.
  • You should be tracking the "freshness" of identifiers through some other system, and you should have some way to exclude identifiers from the refresh upload if these identifiers have been abandoned (are not expected to be seen again in serving).

Processing time

Expect to wait up to 24 hours from the time of batch upload to the time when all batch changes are reflected in serving. In other words, upload your files at least 24 hours ahead of when you need the segments in serving.

Additionally, expect to wait up to 48 hours from the time of batch upload before the changes are visible in the Ad Manager user interface.