Technical

Tips, Tricks and Technical Information

Net-X-Copy - Partial File Restore

verifycc icon

Net-X-Copy is part of the Net-X-Convert, Net-X-Proxy, and Net-X-Code server software. It is designed to convert audio/video files from one type to another and to perform partial file restores from indexed and non indexed files on disk, tape or cloud. This article contains the Linux/macOS/Windows command line API for this tool.  A RESTful API is also available.

Net-X-Copy - Partial File Restore

Net-X-Copy supports partial file restore for all the standard broadcast and post production formats we support without the need for a mezzanine or intermediate file format.  The original files are indexed, a time code accurate proxy is created, and then any section of the original file can be recalled via time code or absolute offsets based on HTML5 playback of the proxy.  The high resolution asset can be stored on near line storage, cloud storage (HTTPS/FTP/Amazon/Google/SMB), tape or any other kind of block, object or file storage your server can access.

There are four modes the partial file restore can operate in:

  1. Full online:  This is the simplest mode where access to the source file is fast.  Here a call can be made with absolute offset values or time code values to determine the area to restore, and the header/metadata/index in the main file will be used to find the audio/video data and produce the newly wrapped/restored file in the same format.
  2. Live online:  For live files like MXF Open, transport streams or other growing files, an area can be clipped or restored while the file is still writing.  This includes local, network and cloud sources.  Here the header/metadata is pulled from the file and the index is calculated based on absolute or time code values to find the media for the new file.
  3. Slow online:  For files that are stored on very slow storage, like Amazon's Glacier or other cold storage systems, an index for the file should first be generated.  The file is pushed to long term storage, and the small index file is kept locally available.  To restore a clip, the index file is used to calculate the minimum amount of bytes required to create the new file.  This byte range can then be retrieved from the file during the restore by Net-X-Copy.
  4. Full offline.  When the file is completely offline, using tapes, stored hard drives or other methods, an index must be generated before the file is stored.  This index can then be used to calculate the minimum amount of the file (as a byte range) that needs to be retrieved to create the restore/wrap file.  Once this byte range is available on temporary local storage, the actual restore is called to make the new file.

The index/proxy/restore can be controlled via our RESTful API in www.net-x-code.com or via the command line parameters described below.


Typical Scenarios

This section has some typical proxy, convert and partial file restore workflows. The first scenario is a complete list. The subsequent scenarios assume the first few steps in the first scenario have already been accomplished

Scenario 1 - full access/on line source

  • The file (MXF, MOV, AVI, CINE, etc) arrives at ingest
  • A “command=copy&profile=index” command is sent to index to original filename
  • A “command=copy&profile=mp4-h264” command is sent to make a proxy file with time code, multi-track audio, closed captions, metadata and proxy index filename
  • Two “command=copy&pisrc&pidst” commands are sent to create JPEG images for the source and proxy files
  • At this point, the proxy and main index can be stored in a real or near real time storage, and the main file may be moved to long term storage, tape, cloud (google/s3) or other offline storage
  • The user uses the HTML5 player’s time code (or other time code source) to set one or more In and Out points on the file that needs to be restored
  • A “command=copy&profile=wrap” is sent to access the bytes of the original file and create a new file of the same type, without any recompression of audio or video, at the target location

Scenario 2 - tape restore

  • This assumes the basic processing in Scenario 1 has been done
  • A “command=getcopyinout” is sent with the absolute or time code based in and out points, and the index of source file it will come from
  • This returns a series of one or more file names with start and end byte locations
  • At this point, the controller restores those byte areas of the files to the name specified by the return
  • Once the areas are restored, a “command=copy&profile=wrap” is sent along with the temp folder to create the new output filename

Scenario 3 - cloud restore

  • This assumes the basic processing in Scenario 1 has been done
  • If the index file is stored on cloud, it can be restored locally first, or read directly from the cloud (https, ftps, aws)
  • If the main file is in Glacier, then a command will be sent to restore the section needed to S3 before the restore is done
  • Once there is access to the file, or file part, the “command=copy&profile=wrap” can be called normally
  • If the resource is in available cloud storage (e.g. not Glacier), then partial file restores may be done from the original file without indexing it first. This will cause more data to be read, but only the headers and tables necessary to find the audio/video/data the restoration needs

Scenario 4 - in line conversion

  • This assumes the basic processing in Scenario 1 has been done
  • For any restore scenario, the file restored can be a byte accurate re-wrap of the original into a new container, or it can be translated in process (on the fly) to any supported standard format. These formats include MXF Op1a, Op-Atom, P2, IMX, D11, IMF, MOV, Uncompressed and many other containers, with codecs including JPEG-2000, XDCam, MPEG-2, h.264, HEVC, AVCi 100/200, XAVC-S, XAVC, Long-G, TR-01, DV and many others
  • The commands can also be used with or without index files to convert all our part of local clips to any of these formats

PFR File Best Practices

Different workflows require different ways of saving and restoring files, but there are some general rules that can make it easier, especially when working with tape or other non sequential storage systems. Media files can be roughly broken down into a few categories:

  • Self contained, single files (MXF OP1a, MOV, AVI)
  • Multiple stream files (AVI+WAV, Avid OPAtom, MOV QT Reference)
  • Multiple files per stream (P2 MXF, XDCam MP4, Canon C300/700 MXF)
  • Sequences (DPX, TGA, TIFF)

If you are restoring from reasonable speed, random access devices, then all these types can be simply stored and retrieved as is. If there are sequential access, speed or cost issues, then it makes more sense to make each of these as easily accessible as possible before storing them.

Self Contained

These can be indexed and stored directly.

Multiple Stream Files

These can also be indexed and stored directly, as the RTIN can point at one file per stream. Net-X-Code automatically finds and joins the parts of the streams if it is a supported file type like Avid OPAtom, MOV reference, or video file with rationally named audio. If they are stored this way, there will be one chunk from each file that needs to be restored to make the output file, as the media is in separate files. This is handled in the getinoutbytes return as a series of temp names and start/end byte ranges. If you prefer a single chunk, please follow the guidelines in the Multiple Files Per Stream section.


Multiple Files Per Stream

The RTIN cannot describe streams that have multiple parts per stream. These are normally broken up to get around older disk format restrictions, often at 2 or 4 gigabytes. To deal with these, they should be pre-processed (wrapped) to a self contained file like MXF OP1a before they are stored to tape. Using Net-X-Code to wrap them will cause the original audio/video to be copied to the new MXF without recompression and will automatically generate the RTIN as it is creating the new file. A proxy file can also optionally be created while rewrapping the file. This MXF now becomes the file you would restore from, and that should be stored to tape.
Net-X-Code fully supports automatically joining the parts of most broadcast and post production files for playback and wrapping, including:

  • Panasonic P2
  • IMF
  • AS-02
  • DCP
  • Canon C300/700
  • Sony XDCam MXF
  • Sony XDCam MP4
  • Avid OP-Atom
  • Grass Valley K2 Server Format
  • Multi card MXF capture


Avid OP-Atom – Special Case

Net-X-Code supports automatically joining Avid OP-Atom files and creating Avid OP-Atom files that can be directly dropped into the Avid bin for pickup on the next database refresh. If you are using this workflow, then then the Multiple Stream Files method works best. If you are importing files into Avid then you will not be able to use OP-Atom, as Avid cannot import even its own. It will want to see the same essence, but in an OP1a container. In this case, you should re-wrap the OP-Atom to OP1a before storing them. It is also possible to restore OP-Atom to OP1a using the “type mxf-op1a” when doing the restore, but if they are stored as OP-Atom you will still need to restore a chunk from each stream to PFR them.


Sequences

As sequences can be restored ‘per file’, they do not need to be indexed.


Sample Command Lines

Basic Conversion Re-Wrap

NetXCopy -s <sourcefile.gxf> -t <targetfile.mxf> -p wrap

Create New Sony XDCam 50

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-OP1a-MPEG

Make a Basic Proxy with Audio Mix Down

NetXCopy -s <sourcefile.lxf> -t <proxyfile.mp4> -p mp4-h264 -stereo

Make a Specific Proxy

NetXCopy -s <sourcefile.gxf> -t <proxyfile.mp4> -p mp4-h264 -width 360 -height 202 -h26xprofile main -h26xlevel 41 -gopsize 60 -kilobitrate 800

Make an AS-11 DPP

NetXCopy -s <sourcefile.mov> -t <as11dpp.mxf> -p mxf-as-11-hd-dpp

Make a 10 Bit h.264 OP1a MXF

NetXCopy -s <sourcefile.dpx> -t <mxfop1ah264_10.mxf> -p OP1a_HBR_50

Create An Index For Offline/Tape File

NetXCopy -s <sourcefile.mxf> -p index

Partial File Restore From a Tape

Request the start and end to restore from tape for the actual restore

NetXCopy -s <sourceindex.rtin> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 

This will return (tempFile=sourceindex_147894272_298274816.mxf, a start byte position u64in=147894272, and an end byte position u64out=298274816).  Copy the byte section specified from the source= to the tempFile= into your temp directory (d:\record\).  Then call the actual restore:

NetXCopy-s <sourceindex.rtin> -t <targetfile.mxf> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 -tempFolder d:\record\

Replace Audio and Captions in an MXF File

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-xdcam-1080i -cc <newlanguage.mcc> -afile <newlanguage.wav>

Make A JPG From A Video Frame

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piframe 100

Make a Series Of JPG Frames For Each Second

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piskip 30

Write an XMP Metadata File To Disk

NetXCopy -s <sourcefile.avi> -m

Get Clip Info From a Camera Card

NetXCopy -s <path-to-card> -p cardinfo -t 2


Command Line Parameters

netxcopy: Wrap, Convert, Extract, Thumbnail and Index Media Files (v7.1.0.491)
netxcopy -s <sourcefile> -t <targetfile> [-a <ackfile>] -p <profile> [-in <00:01:00:00> -out <00:02:00:00> -fg]

-s <sourcefile> - The source file name and path
-t <targetfile> - The target file name and path
-a <ackfile> - The ACK file name and path. This is the XML acknowledgement file made after a copy
-p metadata - Return XMP metadata for a media file
-p cardinfo - Return XML/JSON info on clips on a camera card
-p dir - Return XML/JSON directory listing
-p clipfiles - Return all the files associated with a media clip
-p md5 - Calculate or check the MD5 value source, send compare string as target
-p <profile> - Profile to use. Current profiles include:
    > copy - copy the whole file
   > wrap - re wrap file or part of a file
   > index - create an RTIndex for a file
   > caption - #- Extract captions from a file(see types below)
   > cardinfo - cardinfo
   > dir - dir
   > clipfiles - clipfiles
   > md5 - create md5
   > xmp - create xmp
   > picon - create picon
   > metadata - show metadata
   > getCopyInOut - get the extents required for a pfr, or use them with a temp file
   > getFirstAndLastTimecode - get the first and last timecode of a file
   > examine - Show file info
   > mov-YCbCr8Bit - QuickTime MOV YCbCr 8 Bit
   > mov-dvcprohd - QuickTime MOV DVCProHD
   > mp4-h264 - QuickTime MOV h264
   > mxf-xdcam-720p - MXF OP1a XDCam 720p 50 4:2:2
   > mxf-dvcprohd-720p - MXF OP1a DVCPro HD 720p
   > mxf-xdcam-1080i - MXF OP1a XDCam 1080i 50 4:2:2
   > mxf-dvcprohd-1080i - MXF OP1a DVCPro HD 1080i
   > mov-prores422 - QuickTime MOV ProRes 4:2:2
   > mov-proreshq - QuickTime MOV ProRes HQ
   > mxf-OP1a-MPEG - MXF OP1a MPEG 50 4:2:2
   > mxf-OP1a-h264 - MXF OP1a h.264
   > mxf-OP1a-HDF - MXF OP1a HDF 50 4:2:2
   > mxf-as-11-hd-dpp - MXF AS-11 HD DPP
   > scaledown2000k - MP4 - Scaledown to 960x540 with a target bitrate of 2mbits
   > scaledown500k - MP4 - Scaledown to 480x272 with a target bitrate of 500kbits
   > mxf-as-11-sd-pal-dpp - MXF AS-11 SD PAL DPP
   > mxf-as-11-sd-ntsc-dpp - MXF AS-11 SD NTSC DPP
   > hd1080-5000kbs - MP4 - HD 1080 with a target bitrate of 5 mbs
   > hd720-2500kbs - MP4 - HD 720p with a target bitrate of 2.5 mbs
   > hd360-1250kbs - MP4 - HD 360p with a target bitrate of 1.25 mbs
   > h264-7500kbs - MP4 - Any resolution with a target bitrate of 7.5 mbs
   > Proxy-h264-5000kbs - MP4 - Any resolution with a target bitrate of 5 mbs
   > LBR-h264-10000kbs - MP4 - Any resolution with a target bitrate of 10 mbs
   > mxf-OP1a-JPEG2K - MXF OP1a JPEG-2000 4:2:2
   > mxf-AS-02-h264-10 - MXF AS-02 h.264 50 mbs
   > DASH-MP4-Multibitrate - MP4 - DASH MP4 Multibitrate
   > HLS-TS-Multibitrate - TS - HLS Multibitrate
   > TS-TR-01-JPEG-2000 - TS TR-01 JPEG-2000
   > mxf-OP1a_HBR_50 - MXF OP1a HBR 264 10 bit 50mbs
   > mp3-128kbps - MP3 bit rate 128k
   > mp4-XAVC-S_4_2_0 - MP4 Sony XAVC-S 4:2:0
   > mp4-XAVC-S_4_2_2 - MP4 Sony XAVC-S 4:2:2
   > mov-prores444 - QuickTime MOV ProRes 4:4:4
   > mov-proresxq - QuickTime MOV ProRes XQ
   > aces - ACES Image Files
   > dnxhd-mxf-720p - DNxHD 720p 50, 59, 60
   > dnxhd-mxf-1080p - DNxHD 1080p 25, 29, 30
   > dnxhd-mxf-1080i - DNxHD 1080i 25, 29, 30
   > dnxhr-mxf-10-hq - DNxHR 10 Bit
   > dnxhr-mxf-sq - DNxHR SQ
   > dnxhr-mxf-lq - DNxHR LQ
   > TS-MPEG2 - TS-MPEG2 - MPEG-2 4:2:0/passthrough transport stream
   > TS-h264 - TS-h264 - h.264 4:2:0/passthrough transport stream
   > wave - WAVE audio only
   > MXF-RDD-25 - MXF RDD-25 Proxy Writer
   > amt3-LowQuality - MXF Avid Media Low Quality Writer
   > amt3-StandardQuality - MXF Avid Media Standard Quality Writer
   > amt3-HiQuality - MXF Avid Media High Quality Writer
   > amt3-HQX_10 - MXF Avid Media HQX 10 Writer
   > amt3-DNxHD36 - MXF Avid Media DNxHD 1080 Writer
   > MP4-MultiOutput - MP4 - Multiple MP4 Output
   > HEVC-h265-10000kbs - MP4 - HEVC Any resolution with a target bitrate of 10 mbs
   > mov-xdcam - MOV - MPEG-2 50 4:2:2
   > mkv-ffv1 - MKV - FFV1 Lossless
   > dnxhr-mxf-hq - DNxHR HQ
   > mov-proreslt - QuickTime MOV ProRes LT
   > mov-proresproxy - QuickTime MOV ProRes Proxy
   > mp4-vtt-h264 - QuickTime MOV h264
   > kipro-prores-lt - QuickTime MOV ProRes LT
   > amt3-XDCam - MXF Avid Media XDCam Writer
   > amt3-Proxy - MXF Avid Media Proxy Writer
   > amt3-Uncompressed - MXF Avid Media Uncompressed Writer
   > transport_stream - h264 transport stream
   > mxf-xdcam35-1080i - MXF OP1a XDCam 1080i 35 4:2:0
   > mxf-xdcam35-1080p - MXF OP1a XDCam 1080p 35 4:2:0
   > mxf-xdcam35-720p - MXF OP1a XDCam 720p 35 4:2:0
-in <00:01:00:00> - the starting point for the output file in time code or absolute position
-out <00:02:00:00> - the ending point for the output file in time code or absolute position
-absin <200> - the absolute (zero based) start time for the output file (overrides -in)
-tcoffset <00:01:00:00> - Offset the timecode by this amount
-tc <01:00:00:00> - Replace the output timecode starting with this timecode
-ub <ABCD0123> - Replace the output userbits with these userbits
-absout <400> - the absolute (zero based) end time, exclusive, for the output file (overrides -out)
-width <width> - output width of the video (only for arbitrary codecs like h264, hevc and prores)
-height <height> - output height of the video (in not set, the input size or codec size will be used)
-copy - make a copy of the file section we need, instead of reading directly
-dest - folder or folder and file name for the temp file when using copy
-dest <folder> - folder or folder and file name for the temp file when using copy
-cc <mcc/scc file> - replacement closed caption file>
-afile <path-audio-file> - replacement source audio track<s>
-v <path-video-file> - replacement source video track
-stereo - force a stereo pair (mix down) output
-aroute <12345678/auto> - route channels to specific outputs
-uuid <uuid string> - override the UUID of the file with this one
-kilobitrate <kbit rate> - override the kilo bit rate
-h26xprofile <baseline / main / high / high10 / high422 / high444> - override the profile type
-h26xlevel <51> - override the level
-encodemode <0 / 1> - 0 normal, 1 fastest
-gopsize <15> - size of encoded gop
-scenedetect <0> - aggressiveness of scene detection (0-100)
-tempfolder - Temporary folder to store partial file
-alignment - Alignment value for any temporary partial files, for GetCopyInOut profile
-flags flag - Extra flags for special operations
   allSubAudioTracks, aiProfileID-##, audiofirstonly, audioforce8channels, ccExtractChannel-##,
   ccOutputDirectory-**, convert708to608, use708CompatibiltyBytes, deinterlace-all, deinterlace-auto,
   deinterlace-sd, deinterlace-hd1080, forceinterlaced, forceprogressive, deinterlace-motionadaptive,
   deinterlace-yadif, deinterlace-edge, deinterlace-triangle, deinterlace-fieldduplication,
   eMamAssetID-##, eMamCategory-**, eMamProxyID-##, extractCCLine21, fixOut, forceCCVidStream, forceCaptions,
   forceAudioMono, forceAudioStereo, forceAudioBlock, forceOutputFramerate-##, fpsCompensation-##,
   frameAccurate, interplay, interpretFileEncoding-##, markFirstTime, multi, nocache, notifyeMam,
   offsetTC-##, overrideTC-##, outIsDuration, proxyAfterRecord, overrideextension, proxyDisable, proxyEnable,
   proxyRecord, rtinReplaceSource, sampleaccurate, shuffleAudio, testMode, processdiscontinuities,
   tctype24/25/df/ndf/50/59/60, timeConversion, timeIsMs, use2channelsonly, zeroBasedOutput
-m - Save the metadata in an XMP file
-fg - force the GUI on
-fc - force command line
NOTE: the parameters in [square brackets] are optional.

For JPEG picons/thumbnails
-pisrc <file-to-picon> - source for the image
-pidst <where-to-make-picon> - target folder and name
-pisize <size> - size of picon, 100%
-piframe <frame-offset-into-file> - frame to use to make the picon
-piskip <number-of-frames-to-skip> - if set, make a picon of each frame at this distance for the whole file
-width <width> - output width of the picon image

For Caption Extraction, these are the numbers to use with -p caption-#
> caption-# - eg caption-0=mcc, caption-6=SCCAUTO, etc
0=MCC, 1=AAF, 2=TEXT, 3=SCC, 4=SCCDF, 5=N0, 6=SCCAUTO,
100=Avid Caption, 101=Belle Nuit Subtitler,
102=CapMaker Plus, 103=Cheetah Caption, 104=Csv, 105=Csv2, 106=Csv3,
107=D-Cine SMPTE 2007, 108=D-Cine SMPTE 2010,
109=EBU STL, 110=Final Cut Pro Xml,
111=Final Cut Xml Gap, 112=Final Cut Pro X Xml,
113=Final Cut Pro XCM, 114=Flash Xml,
115=iTunes Timed Text, 116=JSON, 117=JSON Type 2,
118=JSON Type 3, 119=JSON Type 4, 120=QuickTime text,
121=SAMI, 122=SAMI modern, 123=Scenarist, 124=DVD_SCC,
125=DVD_SCCDF, 126=SoftNi sub, 127=Sony DVDArchitect,
128=Sony DVD Exp dur, 129=Sony DVD line/dur,
130=Sony DVD Tabs, 131=SubRip, 132=Sub Station Alpha,
133=Swift V2, 134=Timed Text 1.0, 135=TT 2006-04 .dfxp,
136=TT 2006-04 .ttml, 137=TT 2006-04 .xml, 138=WebVTT,
139=YT Annotations, 140=YouTube sbv,
141=YouTube Transcript, 142=D-Cinema interop,
143=WebVTT-webvtt, 144=SMPTE-TT 2052,
145=Netflix Timed Text,
-q Quiet mode.
-l Loud mode.
-r 01:00:00:00 Override timecode start
-R -00:30:00:00 Offset timecodes (e.g. subtract half an hour)
-0 Zero based timecode output
-1 Mark this first timecode in file
-h # eMAM source clip ID
-s Force CC in video stream
-X Extract CC from line 21 (SD only)
-M # CC channel to extract (def: 1)
-C Use 608 compatibility bytes (not 708)
-c # Convert closed captioning to
-f # Force output frame rate (23, 24, 25, 29, 30)
-g # Compensate conversion (2324, 2423, 2930, 3029)
-o <out-dir> CC output directory
-E # Interpret input file using encoding
0=Auto, 1=UTF-8, 2=UTF-16, 3=IBM-850, 4=IBM-860, 5=Windows-1252

 

 

More great products from Drastic

Setting up Google Cloud (OAuth 2.0) Access
Setting up Google Cloud (OAuth 2.0) Access All of Drastic's version 6 or greater software supports authenticating via OAuth 2.0 across an https transport. This is most useful for Net-X-Code's partial fil... Tips, Tricks and Technical Information
Using FlowCaster with Avid Media Composer
Using FlowCaster with Avid Media Composer FlowCaster creates an Open I/O board in Avid creative software like Media Composer.  As a virtual video board, it can send the same high quality audio... Tips, Tricks and Technical Information
OpenMXF support
OpenMXF support Drastic software supports the OpenMXF specification extension to the SMPTE MXF specification (SMPTE 377M). The advantage of OpenMXF is that while the file is st... Tips, Tricks and Technical Information
Simple Quality Measurement (MRAnalyze)
Simple Quality Measurement (MRAnalyze) MediaReactor includes tools it uses to automatically check QA translations using industry standard metrics.  This tool, called MRAnalyze, can be used ... Tips, Tricks and Technical Information
Java on Dual Monitors
Java on Dual Monitors Java on Dual Monitors - NVidia vs. Java.  There are setups required on some systems to allow QuickClipXO and VTRIF to display full screen on the second DVI... Tips, Tricks and Technical Information
Image

For more than two decades, Drastic™ has been developing cutting edge solutions for television, post production and sports broadcasting, from real time web delivery to 8K broadcast.

We offer standalone software for the end user or enterprise, integrated solutions for automated workflows, and OEM tools for custom applications or branded devices.

Contact Us

Address:
523 The Queensway, Suite 201
Toronto, ON
M8Y 1J7, Canada

Phone: +1 (416) 255 5636

Email: info@drastictech.com

Fax: + 1 (416) 255 8780