Customer Sign In

upLynk

Slicebot: Automated ingestion workflows

Overview

The Slicer is the tool used to send content to the encoder cloud and comes in GUI and command line versions. In both cases, the slicer is manually invoked to kick off the process of preparing content for encoding. In order to support completely automated workflows, the command line versions of the slicer comes with a companion tool called Slicebot.

The slicebot tool monitors one or more directories on your filesystem and watches for files to arrive. New files placed into a monitored directory are automatically sliced and sent to the cloud. This guide explains how to configure and use slicebot to automate content preparation.

Running Slicebot

You create a configuration file to tell slicebot where to look for files and how to process them. The config file format is described in the Controlling Slicebot section, below. Once your config file is ready, you start slicebot like this:

./slicebot /path/to/your/config/file 

Run slicebot without any parameters to see some additional options. For example, the --dry-run option is a useful way to test your configuration parameters without actually processing any files:

./slicebot --dry-run slicebot.cfg 

Normally slicebot runs indefinitely until it is interrupted or it encounters a fatal error, such as a problem with your config file settings. When slicebot terminates in this way, its exit code is non-zero.

If you need to shut down slicebot, but do not want to interrupt any slicing jobs in progress, send the slicebot process the SIGINT signal:

kill -SIGINT 1234  # use slicebot's PID 

This will tell slicebot to finish its current slicing job, if any, and then to shutdown without processing any additional files.

Controlling Slicebot

The slicebot config file is a simple text file created with any text editor; when you run slicebot, you specify the path to this config file. The config file consists of one or more sections, within which are listed settings in the form of name=value:

Example slicebot config
[global]
enc_wait = 1
mail_host = smtp.gmail.com:587
done_emails = sys@mycompany.com, me@mycompany.com
fail_emails = me@mycompany.com

[~/t/slice/a/]

[~/t/slice/b]
enc_wait = 0
username = me@email.com
password = xyz123

The available settings are outlined in the Configuration Settings section below.

Each section header is the full path to a directory you want slicebot to monitor for files, and the settings inside the section are applied to any files located in that directory. A special section named "global" can be used to specify default settings to use for all files in all directories.

Using the example config above, if slicebot encounters a file called ~/t/slice/b/movie.mp4 and needs to know what value to use for the enc_wait setting, it will use 0 since that is the value from the ~/t/slice/b section of the config. If the slicebot later encounters the file ~/t/slice/a/other.mp4, it will use a value of 1 for enc_wait - it will first look in the ~/t/slice/a section of the config and, not finding it there, will look in the global section instead.

In some cases it is useful to specify some settings on a per-file basis. For example, a video's description pertains to a specific video, so it wouldn't make sense to list that value anywhere in the config file. For these cases, you can create a small text file for an individual file, and place it in the same directory as the media file. Within this file you can put whatever config settings you want applied to that specific file.

Per-file configs must have the exact same name as the media file plus the ".cfg" (all lowercase) extension. For example, if you wanted to specify the description for a video file named film.mp4 in the /tmp/foo directory, you would create a text file named /tmp/foo/film.mp4.cfg and within it place a single line:

description = This is my new film

Notice that the per-file config has no section header in it, since it applies to only one file and not a directory of files.

Settings in per-file configs override settings in a directory section in the slicebot config, just as directory section settings override any settings in the global section of the slicebot config.

A best practice when using per-file configs is to create the config in the monitored directory first, and then to copy the media file into the monitored directory. The reverse order also works, but placing the files in this order guarantees there will never be case where the media file is processed by slicebot prematurely, before the accompanying config is ready. See also the require_config setting.

Slicebot Configuration Settings

This section describes the supported configuration settings that can be used to control how slicebot processes files. The following settings are specifically for how slicebot behaves.

usernameThe upLynk account username to use when running the slicer. This setting is generally required, but may be omitted if the SLICER_USER environment variable is set.
Example:  username = bob@bob.com
passwordThe upLynk account password to use when running the slicer. This setting is generally required, but may be omitted if the SLICER_PASS environment variable is set.
Example:  password = my_secret
done_directoryThe directory into which the media file should be moved after successfully slicing it. If the directory does not exist, slicebot will attempt to create it. If no done_directory is specified, or if done_directory=none, then once a file has been successfully sliced, it will be erased instead of moved into another directory.
Example:  done_directory = ~/media/out
fail_directoryLike done_directory, except it applies to files that fail to slice properly. If not specified, a file that fails to slice properly is erased.
Example:  fail_directory = ~/media/failed_files
notify_retriesDefines how many attempts will be made to deliver a HTTP notification (i.e., start_url, done_url, or fail_url) before moving on to the next file.
Default value: 3
Example:  notify_retries = 10
notify_retry_waitThe number of seconds slicebot will wait after a failed HTTP notification before trying to reach the remote server again. Default is 10 seconds.
Example:  notify_retry_wait = 30
multibotWhen set to 0, slicebot assumes it is the only slicebot operating on a given directory. Use 1 (the default) to allow multiple slicebots running on separate computers to all process files in the folder simultaneously. Note: all slicebots consuming files from the same folder must run with multibot mode enabled.
Example:  multibot = 1
max_slicesLimits the number of temporal slices to generate from the media file, instead of slicing the entire file. Useful for troubleshooting or slicing short segments of media. Each temporal slice is a few seconds long. The given number is an approximation; a few additional slices may be generated.
Example:  max_slices = 20
allow_inlineIf true or 1 (the default), slicebot will look for and process any inline settings in media filenames. If false or 0, any inline settings will be ignored and files will be processed normally.
Example:  allow_inline = 0
helper_noextIf true or 1, slicebot will not include the source filename's file extension when looking for helper files (such as per-file configs or TTML files). For example, with a source file of movie.mp4, slicebot would normally check to see if per-file settings were in a file named movie.mp4.cfg. With this setting in effect, however, it would instead check for movie.cfg.
Example:  helper_noext = 1
ignoreApplies only to config sections. If true or 1, this section is not monitored by slicebot. This setting is an easy way to temporarily disable monitoring a section from your config file.
Example:  ignore = 1
timedmeta<ms>:<key>=<value>[,...] Adds timed metadata at the time indicated by <ms>.
Example:  timedmeta = 0:TIT2=MyTitle, 2000:TCOM=Myself
mix_atracksMixes audio tracks. Useful for cases where the source audio has e.g. separate left and right audio tracks instead of just using a single audio track with stereo audio. Value is a comma-separated list of audio track numbers to channel abbreviations, where 0 denotes the first audio track in the source file.
Example:  mix_atracks = 0=FL,1=FR,2=C,3=RL,4=RR,5=LFE
halt_on_errorIf true or 1, slicebot will stop processing if a job fails due to either a problem with the media file or a problem with the job settings. By default, slicebot will move on to the next job; this setting can be useful for initial setup or troublesohoting. Must be specified at the global or directory level.
Example:  halt_on_error = 1
ignoreSuffixIgnores the any file with the given suffix (case insensitive). You may provide a comma delimited list for multiple suffixes.
Example:  ignoreSuffix = db,txt,jpg
ignoreNameIgnores the exact filename (case insensitive). You may provide a comma delimited list for multiple files.
Example:  ignoreName = thumbs.db,.DS_Store

Slicer Configuration Settings

The following settings are specifically for how the slicer behaves when invoked by slicebot.

auto_replaceIf set to true or 1, any external ID supplied will be prepended with '_replace:' if not already present.
Example:  auto_replace = 1
autoexpireOnce the asset is encoded, marks it for auto-deletion after the specified number of hours.
Example:  autoexpire = 48 # the system will erase this asset 48 hours after encoding is done
breaksOne or more time ranges, in seconds, that should be marked as ad breaks. Original content is removed and mark points are stored in the database for use with ad-supported live or VOD playback.
Example:  breaks = 10.1-30.58,105-151.332
bugAdd an overlay to the video. Specify the following bug parameters: <filename>,<start time>,<duration>. Start and duration are specified in units of miliseconds. Start may be a negative number to indicate that it should start the given number of milliseconds from the end of the asset. To specify multiple bugs, separate them with 3 commas.
Example:  bug = overlay1.png,0,5000,,,overlay2.png,-5000,5000
This will add overlay1.png at the start of the asset for 5 seconds, and overlay2.png at the end of the asset for 5 seconds. If multiple bugs overlap, Z-order is determined by list order. In this case overlay2.png would appear on top of overlay1.png
descriptionA description of the media file. Generally not used outside of a per-file config.
Example:  description = Holiday office party, 2009
done_emailsA comma-separated list of email addresses that should be notified each time a file is successfully sliced. Note: this setting requires at least the mail_host setting.
Example:  done_emails = fred@gmail.com, george@yahoo.com
done_urlDefines the URL to which a HTTP POST request will be submitted whenever a file is successfully sliced.
Learn more.
Example:  done_url = http://myserver.net/myservice
enc_waitWhen set to 1, the slicebot will wait until both slicing and encoding are complete before moving on to the next file. By default, slicebot moves to next file once slicing is complete and lets the cloud-based encoding finish in parallel.
Example:  enc_wait = 1
external_idAn ID to store with the encoded media, used to tie an asset in the upLynk system to e.g. an ID in your own database. If the external ID is prefixed with '_replace:', once the file is fully encoded, any existing assets with the same external ID will be erased.
Example:  external_id = ep20115e_12
Example:  external_id = _replace:ep20115e_12
fadeFade in/out audio and/or video. Specify the following fade parameters: <in|out>,<audio|video|both>,<start time>,<duration>. Start and duration are specified in units of miliseconds. Start may be a negative number to indicate that it should start the given number of milliseconds from the end of the asset. To specify multiple fades, separate them with 3 commas.
Example:  fade = in,both,0,5000,,,out,both,-5000,5000
This will fade audio and video in at the start of the asset for 5 seconds, and out at the end of the asset for 5 seconds.
fail_emailsLike done_emails, but the list of email addresses to be notified when a file fails to slice. Note: this setting requires at least the mail_host setting.
Example:  fail_emails = ops@company.org
fail_urlDefines the URL to which a HTTP POST request will be submitted whenever a file cannot be sliced.
Learn more.
Example:  fail_url = http://myserver.net/myservice2?status=fail
force_aspect_ratioThe aspect ratio to force in the format num:den
librariesComma-separated list of shared libraries into which the new asset should be placed. Each item is the name or GUID of a shared library you own. Using this setting has the same effect as manually adding the new asset to one or more of your libraries in the CMS. Note: if you use the library name, it must exactly match the name in the CMS (including case). For this reason it is generally better to use the library GUID.
Example:  libraries = 35700c786f2b4f03abb880d2b0bf70fb, 8b085b99d89a4617969682fc23757da2
mail_fromThe "From:" email address to use when sending emails.
Example:  mail_from = automation@company.org
mail_hostThe host (and, optionally, port) of the mail server to use for sending email. Mail servers often use 25 for standard email and port 587 for secure email. Use the form host:port to specify the port.
Example:  mail_host = smtp.company.com:587
mail_passwordThe mail account password to use when sending emails; used for mail servers that require a username and password.
Example:  mail_password = g1bb3r1sh
mail_usernameThe mail account username to use when sending emails; used for mail servers that require a username and password.
Example:  mail_username = robo-emails@bob.com
metaAdds one or more string metadata values to the new asset. Use <key>=<value> form. The key should be a lowercase identifier (no spaces or punctuation). To specify multiple key-value pairs, separate them with 3 commas. See also metadata files .
Example 1:  meta = rating=TV-13 # sets the metadata key 'rating' to the value 'TV-13'
Example 2:  meta = rating=TV-13,,,air_date=20130105 # same, but also sets 'air_date' to '20130105'
meta_intAdds one or more integer metadata values to the new asset. See comments above about format and rules for keys.
Example 1:  meta_int = is_ad=1 # sets the metadata key 'is_ad' to the value 1
Example 2:  meta_int = is_ad=1,,,year=2013 # same, but also sets 'year' to 2013
posterSet the poster image to the video frame at the specified time (milliseconds)
Example:  poster = 5000
poster_fileUse the specified file as the poster image for the new asset
Example:  poster_file = poster.png
poster_sizeOverride the default poster image size (128x128) with the specified size
Example:  poster_size = 512x512
require_captionsIf set to true or 1, slicebot will reject a file unless there is a corresponding closed captions file. The file must be in the same directory as the video file and must have the same filename plus a .ttml extension.
Example:  require_captions = 1
require_configIf set to true or 1, slicebot will not process a file unless it has a per-file config. Not valid in per-file configs.
Example:  require_config = 1
skip_drmWhen set to 1, the content is still encrypted but playback is not restricted - a valid playback session is not required to initiate playback. This is useful for ads that need to be played independently and outside of a playback session.
Example:  skip_drm = 1
start_urlDefines the URL to which a HTTP POST request will be submitted whenever slicing is initiated on a file.
Learn more.
Example:  start_url = http://myserver.net/myservice
thumbnailsA comma-separated list of additional preview thumbnails to generate beyond the default set. Each item in the list is a filename prefix to use on the thumbnails follwed by the maximum width and height to use for the thumbnails (the dimensions are a bounding box inside which the thumbnail will fit; the video aspect ratio will be preserved automatically). Note that use of this feature may incur additional encoding and storage costs, and each additional set of thumbnails can affect the speed at which content is processed; use sparingly with live encoding jobs.
Format is <prefix>:<w>x<h>[, ...]
Example:  thumbnails = tiny:50x50,med:400x400
trimTrims some video off the start, and optionally the end, of the file. If a single number is given, it is the amount of video to trim off the start of the video. If two numbers are given (separated by a comma), the second value is the desired final duration of the video after the start and end have been trimmed. Both values are in milliseconds. Note that trimming may not be frame accurate depending on the file type, but should always be within a few milliseconds.
Example 1:  trim = 5000 # remove first 5 seconds of video
Example 2:  trim = 3000,52500 # remove first 3 seconds of video and trim any off the end so that the total duration is 52.5 seconds

This section describes the supported configuration settings that can be used to control how slicebot processes files.

Although most settings can be used anywhere (global section, directory sections, or per-file configs), you will find that some settings are more naturally used in some places more than others. For example, the global section will typically hold things like username, password, and email settings, although they are legal in other places as well.

Setting values support basic value substitution:

  • [FILENAME] - replaced with the current source filename, minus the path. Example, with a source file of /mnt/files/ad.mp4:

    external_id=foo[FILENAME]bar
    would become
    external_id=fooad.mp4bar

  • [FILENAME_SHORT] - replaced with the current source filename, minus the path and file extension. Example, with a source file of /mnt/files/ad.mp4:

    external_id=foo[FILENAME_SHORT]bar
    would become
    external_id=fooadbar

Inline Settings

Most slicebot settings are stored in the slicebot config file (in the global section or directory-specific sections) or in a file-specific config file. In some cases, such as integrating with an existing media production workflow, it may be difficult or impossible to modify the legacy workflow to generate per-file config files. In these situations, it may be helpful to use inline settings, which are slicebot config settings in the media filename itself.

To use inline settings, the filename is changed from the file's name and file extension to be the file's name followed by one or more config settings, followed by the file extension. The file's name and each setting is separated by a caret (^) symbol. For example:

my_movie.mp4  (original name)
my_movie^max_slices=3^external_id=1FA1272b.mp4  (filename with inline settings)

As usual, any supporting files (such as a captions file or a per-file config) must match the filename in order to be detected by slicebot. For example, a closed captions file to accompany the above MP4 file would be named:

my_movie^max_slices=3^external_id=1FA1272b.mp4.ttml 

For convenience, any plus (+) characters are replaced with spaces:

my_movie^description=Standard+Trailer.mp4   (description will be "Standard Trailer")

In the event of a settings conflict, inline settings override any other settings. For example, if section in the slicebot config file had the setting max_slices=10, the above file would be processed with max_slices=3 because the inline setting value of 3 takes precedence.

Care should be taken to avoid creating filenames longer than what is allowed by your operating system. If excessively long filenames are needed, standard config files should be used instead of inline config settings.

Closed captions

You can provide slicebot with closed caption information by placing closed caption data in a TTML file in the same directory as the media file. TTML files are simple text files that can be created with third party tools or with any text editor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="utf-8"?>
<tt xml:lang="en" xmlns="http://www.w3.org/2006/04/ttaf1" xmlns:tts="http://www.w3.org/2006/04/ttaf1#styling">
  <head>
    <styling>
      <style id="defaultSpeaker" tts:fontSize="24" tts:fontFamily="Arial" tts:fontWeight="normal" tts:fontStyle="normal" tts:textDecoration="none" tts:color="white" tts:backgroundColor="black" tts:textAlign="center" />
      <style id="defaultCaption" tts:fontSize="24" tts:fontFamily="Arial" tts:fontWeight="normal" tts:fontStyle="normal" tts:textDecoration="none" tts:color="white" tts:backgroundColor="black" tts:textAlign="center" />
    </styling>
  </head>
  <body style="defaultCaption" id="thebody">
    <div xml:lang="en">
      <p begin="00:00:01.420" end="00:00:02.620"><metadata ccrow="14" cccol="3" /><span tts:fontStyle="italic">THIS IS SOME TEXT...</span></p>
      <p begin="00:00:02.620" end="00:00:03.100" tts:textAlign="left"><metadata ccrow="13" cccol="0" />SPAN<br />MULTIPLE LINES</p>
      <p begin="00:00:06.620" end="00:00:09.790"><metadata ccrow="13" cccol="1" />(Doorbell rings)</p>
      <p begin="00:00:11.100" end="00:00:13.000"><metadata ccrow="14" cccol="8" />♪ LA LA LA SINGING ♪</p>
      <p begin="00:00:20.460" end="00:00:22.170"><metadata ccrow="14" cccol="0" />THIS IS MORE TEXT</p>
    </div>
  </body>
</tt>

The TTML files use a naming convention similar to config files: take the name of the associated media file and add a ".ttml" extension to the filename. For example, if slicebot is processing a file named /tmp/foo/film.mp4, it will look to see if there is a file named /tmp/foo/film.mp4.ttml and, if so, use it for the source of closed caption data. If no corresponding TTML file exists, the video will not have any closed caption data. Note that the helper_noext setting can alter the naming convention for TTML files.

Only hh:mm:ss.sss time formats are currently supported in the TTML file.

Metadata files

You can assign metadata key-value pairs to an asset using individual settings; for larger amounts of metadata, however, this can be cumbersome, so it may be more convenient to make use of a metadata file. This file is a JSON text file that has the metadata you want associated with the media file. As with caption and config files, the name of the metadata file is the media filename with an additional extension: .meta.json. For example, if a media file is named video123.mp4, then an associated metdata file for it would be named video123.mp4.meta.json.

The contents of the metadata file must be a valid JSON object, or processing the file will produce an error. Structured metadata is supported, although presently it is not fully displayed or editable in the CMS UI.

1
2
3
4
5
6
{
    "genre" : "comedy",
    "network" : "ZNN",
    "season" : 1,
    "episode" : 13
}

In general, a given media file will use individual metadata settings or a metadata file, but not both. However, in the event that both are specified for a particular file, metadata from a file is applied first, followed by any metadata settings.

HTTP notifications

HTTP notifications provide an additional way to integrate a system with your content production workflow. This type of notification allows Slicebot to submit a HTTP POST request whenever one of the following events takes place:

  • Slicing Starts (start_url)
  • Slicing Completes (done_url)
  • Slicing Error (fail_url)

Enable HTTP notifications by setting one or more of the above settings to the URL that will be requested.
Learn more about these configuration settings.

HTTP POST Request

Slicebot will submit a HTTP POST request using the default Internet media type (i.e., application/x-www-form-urlencoded). The body of this request will consist of a single line with the following parameters:

Parameter Description
filenameIndicates the full path and the filename of the file.
asset_idIndicates the system-generated unique ID of the new asset.
external_idIndicates the external ID defined in the configuration. If one has not been defined, then it will be set to an empty string.
descriptionIndicates the file's description defined in the configuration. If one has not been defined, then it will be set to an empty string.
statusIndicates the result of the slicing operation. Valid values are either "ok" or "error."

In addition to the above parameters, any custom query string parameters defined in the configuration setting (i.e., start_url, done_url, and fail_url) will be automatically stripped from the request and inserted into the request body. This allows custom parameters to be passed to the remote server as needed.
View a sample scenario.

Note: Custom query string parameters whose names match one of the above system-defined parameters (e.g., filename or description) will be ignored.

Note: To allow for future expansion, the server process that receives the HTTP POST should silently ignore unrecognized parameters.

Notification Issues

Slicebot will log an error under the following conditions:

  • It has been configured to request an invalid URL.
  • A valid URL has been provided, but the server is unreachable.
  • A valid URL has been provided, but it generates a non-200 response.

After which, it will reattempt the HTTP notification several times as determined by the notify_retries setting. After each attempt, it will pause for a few seconds as determined by the notify_retry_wait setting. If it is unable to successfully submit the HTTP POST request, then it will move on to the next file.

Sample HTTP Notification Scenario

This sample HTTP notification scenario assumes that the done_url setting is configured to:

http://www.example.com/slicing/handler?custom=1234&other=5678

The following sample HTTP POST request will be submitted once a file called "video.m4v" has been successfully sliced.

1
2
3
4
5
6
7
8
9
POST /slicing/handler HTTP/1.1
Host: www.example.com
Accept-Encoding: identity
Content-Length: 161
Content-Type: application/x-www-form-urlencoded
Connection: close
User-Agent: upLynk-slicebot/1.0

status=ok&other=5678&description=The%20file&filename=%2Fopt%2Ft%2Fmyvideos%2Fvideo.m4v&external_id=dummy_id&custom=1234&asset_id=a9119d7afc5242319d028794ae283ea7 

Notice that the custom query string parameters (i.e., custom and other) defined in the done_url setting were automatically stripped from the request and inserted into the request body.