Customer Sign In

upLynk

Log file delivery

Overview

The upLynk system generates various reports for encoding, storage, and playback of your content. These reports give you a high level view of how we calculate your bill and how and when your content is being consumed. In some cases, however, you may need more granular information or customized reporting. In this case, you can enable log file delivery or log pushing and the upLynk system will send you lower level, more detailed usage logs. You can use these logs to extract very customized and fine-grained information about how you and your end-users interact with the upLynk system.

The log files contain information about various events that have occured. For example, an event is fired any time a new asset is sliced, or any time one of your users begins playback of an asset or a live channel. An internal process gathers these events from across the entire distributed upLynk system and aggregates them into log files that are then delivered to you for whatever custom processing and analysis you need to do.

Log files are delivered to an Amazon S3 bucket of your choosing. Amazon S3 was selected because it provides a known, stable interface and it is highly scalable, highly available, and has virtually unlimited storage capacity. Sending log files to S3 (instead of e.g. an FTP server) makes the log pushing process more robust while also eliminating many of the IT security concerns that would be otherwise involved.

Enable log pushing

Because logs are pushing to Amazon S3, in order to use log pushing you need to have an S3 account and bucket into which upLynk will place the logs. Please consult Amazon's Getting Started Guide for more information on creating an S3 account and bucket. Do not use an S3 bucket that is publicly readable. Use of a publicly-readable S3 bucket may be cause for termination of your account.

In order to push logs to S3 on your behalf, you need the following pieces of information:

AWS Access Keyan Amazon-assigned unique identifier for your account
AWS Secret Access Keyan Amazon-assigned secret key for accessing your account
S3 Bucket Namethe name of the S3 bucket into which you want upLynk logs to be placed
Key prefixthe path prefix for the log files (essentially a subdirectory name, e.g. "uplynk_logs")

The above information can then be entered on the Advanced tab of the CMS (under the Log Pushing subtab).

The AWS Management Console provides a simple, web-based approach for creating S3 buckets, creating IAM sub-accounts, and so forth.

Once log pushing is enabled, log file pushing will begin immediately and continue until you disable it. Logs are generated throughout each day, so you should expect many small log files as opposed to one large log file per day. By splitting the logs into smaller files, the overall processing latency is reduced - you do not have to wait until the entire day is complete before you begin receiving logs for that day. Note, however, that the distributed nature of the upLynk system means that log entries are not guaranteed to arrive in order. Instead, they are pushed to you as they are aggregated and processed across the entire system.

Log file format

Each push log file is a gzip-compressed plain text file with a unique filename of the form: YYYY-MM-DD-<ownerID>-<unique-random-string>.gz

For example: 2013-02-27-a735c56ea4014685bc74c0a375236cc5-nqbcR78.gz

On December 8, 2017 we will begin sending push log files from our new system which will be using new file names. At first, only a small percentage of push log files will come from the new system, but we will be ramping that up over time until all push log files come from the new system with the new file names. But there will be a significant period where you will be seeing both old and new file names. See below for the file name change. <YYYY-MM-DD>-<unique-id>-<unique-id>PushlogAgg<unique-id>.gz

For example: 2017-12-08-b292bb5eefa54afe8fb20bd733163dff-9d8dPushlogAgg00000294.gz

An example log file might look like this:

Example push log file
#VERSION: 1.0
2011-09-29  16:19:43  slicing_ended   720e497d  1234  0 100
2011-09-29  16:19:20  slice_played    720e497d  76  1  76.13.79.25  Mozilla/5.0%20(Macint...

The first line of the log file tells the format version number. Tools processing push logs may choose to abort if the version listed is higher than what the tool knows how to process. Any subsequent lines that start with a hash (#) character should be ignored as comment lines.

Each line of the file represents a single event - events always fit on one line and there is never more than one event per line. Each line is composed of tab-separated values, and each value has no whitespace. In the event that a value would normally have whitespace, it is "escaped" before being written into the log. For example, the last value on the last line normally would have space characters in it, so those space characters have been converted to '%20' (0x20 being the hexadecimal code for the ASCII space character). Empty values are represented with a dash ('-') character.

The first 3 fields in every line are:

Datethe date the event occurred, in YYYY-MM-DD format.
Timethe time the event occurred, in HH:MM:SS format. Note that this value may be an approximation only, due to latencies in logging and gathering usage information from distributed sources.
Event namethe name of the event that occurred. See the events section for the list of events.

The date and time fields tell when the event occurred in UTC. Robust processing tools should ignore log lines containing an unrecognized event name. As noted previously, log rows are not guaranteed to be in sequential time/date order, although in general events are aggregated to within a few minutes of each other.

The fields after the initial 3 fields vary depending on which event was logged. See the events section for the list of events and their corresponding fields.

New fields may be added to events over time, so log processing tools should either abort on a format version change or gracefully ignore additional fields beyond what was expected.

Logging events

This section lists the various events that can appear in pushed logs, along with the event-specific fields that appear after the event name in the log row.

For events that support custom fields, available objects you can use in custom fields are listed after the standard fields for each event.

slicing_began

A slicing job (live or VOD) was started, resulting in the creation of a new asset.

FieldDescription
job type'live' or 'vod'
asset IDthe unique ID of the new asset
external IDuser-supplied external ID if any, or '-'
channel IDfor live assets, ID of channel in which asset was created, or '-'
descriptionuser-supplied asset description if any, or '-'
slicer software IDtype and platform of slicer software used
slicer versionversion number of slicer software used
event IDIf the slicer is associated with a live event, the event ID will appear here, otherwise there will be a "-" (dash) character.

slicing_ended

A slicing job finished.

FieldDescription
asset IDID of the asset that finished slicing
had errors0 on success, 1 on error
slicesnumber of slices delivered for encoding

asset_play_started

A user started watching an asset. Asset playback starts generate this event regardless if playback happens for VOD or live.

FieldDescription
asset IDthe asset that was played
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
channel IDthe channel ID if the asset start was in a linear channel, otherwise a "-" (dash) character for a VOD asset start
asset is ad1 if the asset was played as an ad or slate, 0 if played as normal content
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system
playing owner IDWhen playing shared content, the owner (account) ID of the account playing the content, a "-" (dash) character otherwise. If the given owner ID is your own account ID, it implies the content being played is from some other account (e.g. an ad or a shared library).
event IDIf the asset was played in the context of a live event, the event ID will appear here, otherwise there will be a "-" (dash) character.

Custom field objects: asset, channel, event, playing_owner

channel_play_started

A user started watching a live channel.

FieldDescription
channel IDthe channel that was played
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system

Custom field objects: channel

channel_ping

A user continued watching a live channel. This event is emitted regularly while a user is watching a live channel and can be used to infer a rough estimate of simultaneous viewers (audience size) during live playback.

Time is divided into ranges or "buckets" of 10 minutes each, and any time a user plays content inside that range of time, a single channel ping event is emitted. Each time bucket has a unique number associated with it. Bucket 0 refers to the epoch (00:00:00 on Jan 1, 1970 UTC), bucket 1 starts 10 minutes after that, and so on.

FieldDescription
channel IDthe channel that was played
asset IDthe asset the user was watching in that channel
bucket numberthe range of time or "bucket" in which some playback occurred
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system

Custom field objects: channel, asset

event_play_started

A user started watching a live event.

FieldDescription
event IDthe event that was played
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system

Custom field objects: event

event_ping

A user continued watching a live event. This event is emitted regularly while a user is watching a live event and can be used to infer a rough estimate of simultaneous viewers (audience size) during live playback.

Time is divided into ranges or "buckets" of 10 minutes each, and any time a user plays content inside that range of time, a single event ping event is emitted. Each time bucket has a unique number associated with it. Bucket 0 refers to the epoch (00:00:00 on Jan 1, 1970 UTC), bucket 1 starts 10 minutes after that, and so on.

FieldDescription
event IDthe channel that was played
asset IDthe asset the user was watching in that event
bucket numberthe range of time or "bucket" in which some playback occurred
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system

Custom field objects: event, asset

slice_played

One temporal slice of media was delivered. Each slice generally represents approximately 4 seconds of media. The timestamp corresponds to the point in time in which the slice was fully delivered to the player.

FieldDescription
asset IDthe asset that was played
slice numberzero-based number of the slice that was played
is live1 if the slice was played in a live content (e.g. live channel), 0 for VOD
user IPthe IP address of the device playing the asset
player user agentthe HTTP "User-Agent" request header sent by the player
player referrer URLthe HTTP "Referer" [sic] request header sent by the player
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
session IDa unique playback session ID generated by the upLynk system
playing owner IDWhen playing shared content, the owner (account) ID of the account playing the content, a "-" (dash) character otherwise. If the given owner ID is your own account ID, it implies the content being played is from some other account (e.g. an ad or a shared library).
channel IDIf the slice was played in the context of a channel, the channel ID will appear here, otherwise there will be a "-" (dash) character.
event IDIf the slice was played in the context of a live event, the event ID will appear here, otherwise there will be a "-" (dash) character.

Custom field objects: asset, event, playing_owner

asset_deleted

An asset was deleted from the system.

FieldDescription
asset IDThe asset that was deleted
causeThe reason the deletion occurred. For example, an asset could be deleted due to user action in the CMS, because the asset was marked to auto-expire, because a slicing job ended with errors, etc.

channel_deleted

A channel was deleted from the system.

FieldDescription
channel IDThe channel that was deleted
causeThe reason the channel was deleted. Channels are deleted only via the CMS, so this field contains the user ID of who deleted the channel, for audit purposes.

event_deleted

A live event was deleted from the system.

FieldDescription
event IDThe live event that was deleted
causeThe reason the live event was deleted. Live events are deleted only via the CMS, so this field contains the user ID of who deleted the live event, for audit purposes.

error

An input error occurred. The system detected an input error of some sort, e.g. an invalid playback URL was presented to the system for playback.

FieldDescription
formatTypespecifies which additional event parameters will be present. In the current implementation, will always be 0. Log processors should not make assumptions about the remainder of the event parameters if a non-zero formatType is encountered.
subTypesystem component or area in which the error occurred. In the current implementation, will always be the string 'web' to denote web services and APIs.
URLthe URL that was requested that generated the error.
user IPthe IP address of the client that sent the URL
HTTP codethe HTTP response error code (e.g. 404) generated as a result of the URL
user agentthe HTTP "User-Agent" request header sent in the request
messagea short description of why an error occurred

session_created

A new session for a viewer is created

FieldDescription
session IDa unique playback session ID generated by the upLynk system.
content IDthe ID for the channel/asset/event that the session was created for.
content typeThe content type for this session. Either asset, channel, or event.
playing owner IDWhen playing shared content, the owner (account) ID of the account playing the content, a "-" (dash) character otherwise. If the given owner ID is your own account ID, it implies the content being played is from some other account (e.g. an ad or a shared library).
user agentthe HTTP "User-Agent" request header sent in the request
euidexternal user ID provided in the playback token, otherwise a "-" (dash) character if none was provided
refererthe http referer for the session
user IPthe IP address of the client that sent the URL

ad_beacon_sent

An event beacon was sent to an ad server. These will include events to 3rd party ad servers. In cases where there are multiple urls for the same event e.g. an ad might have multiple impressions to send to FreeWheel and to various 3rd party ad servers, the group ID field will link urls fired for the same event together.

FieldDescription
group IDLink together multiple urls that are sent to multiple ad servers for the same event e.g. multiple ad impressions for the same ad that is being played.
session IDa unique playback session ID generated by the upLynk system. More details for this session can be gathered from the session_created event
ad asset IDThe uplynk asset ID for the ad that is currently playing. '-' in cases where the event is not tied to an ad e.g. a slot impresssion
event descriptionDescription or type of event from the ad server. Possible values include videoview, slotimpression, defaultimpression, start, firstquartile, midpoint, thirdquartile, complete.
ad break position The break position, either pre, mid., or post, where is mid roll index (zero based) in the content e.g. mid.1 is the second mid roll ad in the content.
ad index in break/pod Index for the ad in the break/pod (0 based) e.g. 1 means the ad is the second ad in the specified ad break. -1 for events that do not have an ad break associated with them
ad IDAd ID from the ad server. For FreeWheel this is the ads/ad@adId. '-' in cases where the event has no ad ID
creative ID (FreeWheel only)ad/creatives/creative@creativeId. '-' for no creative ID
urlThe url for the event that is being fired.
HTTP Statushttp status code returned form the server.

ad_request_error

An error occurred when making an HTTP request to the ad server.

FieldDescription
session IDa unique playback session ID generated by the upLynk system. More details for this session can be gathered from the session_created event.
asset IDThe uplynk asset ID for the content that is currently playing.
ad break position The break position, either pre, mid., or post, where is mid roll index (zero based) in the content e.g. mid.1 is the second mid roll ad in the content.
ad request dataFor FreeWheel customers, this field will contain the smartXML payload that was sent to FreeWheel at the time of the request. All other Ad Platform customers will receive the VAST or VMAP request URL.
error descriptionDescription or error event generated by the exception.
slicer IDThe ID of the slicer used for the currently playing content.
channel IDIf the asset was played in the context of a channel, the channel ID will appear here, otherwise there will be a "-" (dash) character.

Custom log fields

Each event in the log file has a pre-defined set of fields that will always be present for that type of event. In addition to these standard fields, you can define one or more custom fields for some event types. These custom fields can be hardcoded strings or values extracted from items referenced by the event. For example, the asset_play_started event always includes a field that holds the asset ID, but using a custom field you can instruct the log generator to also include the asset's description.

Available objects

Each event definition tells which objects can be used for custom fields. For example, the asset_play_started event allows custom fields to access the asset and channel objects. In other words, when an asset_play_started event is logged, you can define a custom field whose value comes from the properties of the asset being played or the channel being played (if any).

The following tables describe which properties are available for use in a custom field.

asset

Property nameDescription
descAsset description
durationAsset duration in seconds
external_idExternal ID for the asset, if any
audio_only1 if this is an audio-only asset, 0 otherwise
meta.<key name>A particular key inside this asset's metadata
createdWhen the asset was created, in milliseconds since the epoch

channel

Property nameDescription
descChannel description
external_idExternal ID for the channel, if any
meta.<key name>A particular key inside this channel's metadata
createdWhen the channel was created, in milliseconds since the epoch

event

Property nameDescription
descLive event description
external_idExternal ID for the event, if any
meta.<key name>A particular key inside this event's metadata
createdWhen the live event was created, in milliseconds since the epoch

Example

In the following example, three custom log fields have been configured for the slice_played event:

  • asset.duration
  • asset.desc
  • asset.meta.genre
2014-04-22	01:09:43	slice_played	58d31ecee2b9501121e16a16a7b393e8	12	1	71.29.82.30	Mozilla/5.0	-	-	58d31ecee2b9501121e16a16a7b393e	-	-	73.2	test%20asset	comedy
Note that the standard log fields are present, followed by the 3 custom fields. The custom log field names start with 'asset.' because asset is one of the available objects in slice_played_events. Also note that in the case of 'meta', an additional name part is required to identify which metadata key to insert into the field.

Additional notes

  • Custom log fields immediately follow the standard log fields for a particular event.
  • Custom log fields are defined on a per-log-event basis.
  • Custom log fields are formatted the same as other log fields - values are tab-separated and other whitespace is automatically escaped.
  • The upLynk CMS does not currently expose a user interface for configuring custom push log fields. If you need to enable custom log fields for your logs, please contact upLynk support.
  • If you make use of custom fields, you will of course need to make corresponding changes to any tools that process the log files, so that they either ignore the additional fields or handle them properly.
  • If a property is not available in an object, or if the object isn't present for a particular event, the custom field will still be present in the log but will have a value of a single dash character.
  • There is no additional charge for using custom log fields.

Additional notes and policies

Please keep in mind the following when using log pushing:

  • The S3 credentials used for log pushing must have permissions to create and upload objects (write permissions). Use the AWS Management Console to create and test user credentials before supplying them to the upLynk system.
  • In the event that the log pusher is unable to use the credentials you have supplied, any queued log files will be stored and the log pusher will attempt to upload them again later. The system will notify you via email if log pushing failed, but no more than once every 24 hours. If log pushing fails, please disable log pushing or fix the access credentials as soon as possible.
  • Log files that could not be uploaded are not preserved indefinitely, and upLynk cannot guarantee that they will be queued for more than 14 days.Once a log file is erased, the data is lost and the associated events cannot be recovered.
  • Once the upLynk system has successfully pushed a log file to your S3 bucket, the upLynk copy of the log file is erased and connot be regenerated.
  • Please protect the privacy of your users. Some information, such as users' IP address, is sensitive and should be protected. upLynk delivers to you only information that you could have obtained yourself using an in-house media platform, but you should treat all information delivered in push logs with sensitivity. upLynk is not responsible for negligent use or abuse of the information in push logs.
  • Be sure that the S3 bucket to which logs are delivered is not publicly-readable.
  • Playback session IDs are generated by the upLynk system when a user begins playing encrypted content. The session lasts until the user closes their web browser or device playback environment or if the session is inactive for a long period of time. Thus, a single session may represent the playback of multiple assets and/or channels.

Format version history

Over time, the format of the log files is modified as new events and fields are added, and the version number listed in the first line of a log file is adjusted to reflect these changes. Robust tools should gracefully handle changes in versions by raising a proper error, ignorning unknown event types, and ignoring extraneous fields on the end of event log lines.

Version numbers are of the form Major.Minor and, in general, minor revisions only include new event types or new fields added to the end of existing events. Existing messages and their field orders are not modified. In a major version revision, events may be removed, and event fields may be reordered or redefined.

For reference, the format version history is outlined below:

  • 2.13 - Added event_id field to slicing_began event
  • 2.12 - Added event_id field to ad_request_error event
  • 2.11 - Added event_play_started, event_deleted, and event_ping
  • 2.10 - Added event_id field to slice_played event, added event_id field to asset_play_started
  • 2.9 - Added channel_id field to slice_played event
  • 2.8 - Added ad_beacon_sent
  • 2.7 - Added session_created
  • 2.6 - Added error events
  • 2.5 - Dropped slice_delivered and slice_encoded events, added slices to slicing_ended event
  • 2.4 - Added session ID to many events, added sharing oid to asset_play_started and slice_played
  • 2.3 - Added euid to many events, added 'asset is ad' to asset_play_started
  • 2.2 - Removed asset ID from channel_play_started
  • 2.1 - Added channel ID field to asset_play_started event
  • 2.0 - S3 delivery, live channel events
  • 1.0 - First public release
  • 0.9 - Internal development version

Note that when comparing version numbers, each version number component should be compared numerically and not lexographically (e.g. version 3.10 is newer than version 3.9).