implementing a fat based 64 bit dvr

Implementing A FAT Based 64 Bit DVR

Peter VanOudenaren, President, EBS Inc.

DVR System RequirementsDVR System Requirements

Must minimize latencies Must minimize latencies Must minimize CPU utilization.Must minimize CPU utilization. Must have predictable execution timesMust have predictable execution times Must support contiguous files Must support contiguous files Must be DMA friendly Must be DMA friendly

DVR System RequirementsDVR System RequirementsContinued…..Continued…..

Must guarantee volume integrity.Must guarantee volume integrity. Must guarantee session integrity.Must guarantee session integrity. Must monitor resource utilization and Must monitor resource utilization and

block placementblock placement

DVR Application RequirementsDVR Application Requirements

Must support files over 4 GigabytesMust support files over 4 Gigabytes Must continuously collect incoming Must continuously collect incoming

video to a disk buffer.video to a disk buffer.

DVR Application RequirementsDVR Application RequirementsContinued…..Continued…..

Must simultaneously record and Must simultaneously record and display data from the disk buffer.display data from the disk buffer.

Must archive data from the disk Must archive data from the disk buffer.buffer.

Must navigate the disk buffer.Must navigate the disk buffer.

RTFSProPlus Achieves DVR RTFSProPlus Achieves DVR System and Application System and Application

RequirementsRequirements

Extremely fast command executions Extremely fast command executions speedsspeeds

Failsafe III maintains volume integrityFailsafe III maintains volume integrity Failsafe III maintains session integrityFailsafe III maintains session integrity Failsafe III extends the “real time” Failsafe III extends the “real time”

intervalinterval

RTFSProPlus Achieves DVR RTFSProPlus Achieves DVR System and Application System and Application

RequirementsRequirements

Asynchronous operationAsynchronous operation FAT64 64 bit metafilesFAT64 64 bit metafiles Contiguous filesContiguous files Real time readReal time read Real time writeReal time write Real time seekReal time seek

DVR Solutions With RTFSProPlusDVR Solutions With RTFSProPlus

Remote diagnostics toolRemote diagnostics tool•Network attached remote debugger•Real time performance monitoring•Real time monitoring of resource usage •Inspect FAT regions and file chains •Browse file directories•Identify corrupted files•Produce quality assurance reports


No copy extracts of linear segments No copy extracts of linear segments from the circular bufferfrom the circular buffer

Circular file writes and reads process Circular file writes and reads process extract regions appropriatelyextract regions appropriately

Extracted regions appear Extracted regions appear simultaneously in a linear data file simultaneously in a linear data file and the circular data buffer and the circular data buffer


Circular (ring buffer) files for continuous Circular (ring buffer) files for continuous collection of incoming videocollection of incoming video

Separate read and write file pointerSeparate read and write file pointer ““Stream view” file pointersStream view” file pointers ““File view” file pointersFile view” file pointers


API designed specifically for DVR and API designed specifically for DVR and high speed data acquisition applicationshigh speed data acquisition applications

pc_cfilio_open - Create a circular bufferpc_cfilio_close - Close a circular bufferpc_cfilio_read - Read data from circular bufferpc_cfilio_write - Write to circular bufferpc_cfilio_lseek - Seek read or write pointerpc_cstreamio _lseek- Seek read or write stream pointerpc_cfilio_extract - Create an archive file from a section of the circular buffer

pc_cfilio_open – – Open a circular file.


•Select file wrap-around point

•Select 32 bit or 64 bit file

•Select block allocation policy

•Provide extract region buffering

pc_cfilio_read – – Read from a circular file.


•DMA friendly

•Guaranteed to access disk only for file data

•Wraps at file wrap point

•Stops when read catches write file pointer

•Reads extract file regions appropriately

pc_cfilio_write – – Write to a circular file.


•DMA friendly

•Guaranteed to access disk only for writing

file data

•Wraps at file wrap point

•Programmable to stop when it catches read

file pointer

•Overtakes extract file regions appropriately

pc_cfilio_lseek – – Seek in a circular file.


•Maintains separate read an write pointers

•Uses full 64 bit arguments

•Extremely fast, guaranteed zero disk accesses

•File pointers represents a linear view of the

underlying file

•Seeks within extract file regions appropriately

pc_cstreamio_lseek – – CCircular file seek.


•Maintains separate read an write pointers •Uses full 64 bit arguments•Extremely fast, guaranteed zero disk accesses•File pointers represent 64 bit bite offsets in the incoming data stream •Seeks within extract file regions appropriately•Eases mapping time offsets to video content in the buffer

pc_cfilio_extract – – Extract video from the buffer to archive files


•Extract N bytes from the current read file pointer.

•Real time, requires zero data copying in most cases.

•Minimal copying for non-cluster aligned extract

regions

•Establishes soft link between the extract file and ring

buffer

•Extracts to and from either FAT64 files or FAT32 files.

•Allocates clusters to be used when the regions is

overwritten

DVR Programming ExamplesDVR Programming Examples

Creating a circular file Buffering incoming video Displaying buffered video Accessing frames in the buffer Buffering at steady state Archiving buffered video Displaying archived video

The following slides present the source code and run time behavior of typical DVR algorithm implemented with RtfsProPlus.

Creating a Circular FileCreating a Circular Fileint VideoBufferFd;int DisplayFromFd;int ArchiveVideoBufferFd;char *VideoBufferFileName = "dvr.demo";char *VideoBufferDrive = "A:";

/* Create a circular buffer file that wraps at 16 Gigabytes and keeps writing even if the write pointer overtakes the read pointer */

BOOLEAN InitializeVideoBuffer(dword FrameNumber){

dword cluster_size_bytes; EFILEOPTIONS Options; rtfs_memset(&Options, 0, sizeof(Options));

Creating a Circular FileCreating a Circular File Options.allocation_policy = PCE_CIRCULAR_BUFFER;

/* Since the file is over 4 gigabytes, Use FAT64 */ Options.allocation_policy |= PCE_64BIT_META_FILE; Options.circular_file_size_hi = 4; /* 4 X 4 gigabytes */ Options.circular_file_size_lo = 0; /* Wrap point */

/* Force it to be contiguous and preallocate all clusters*/ Options.allocation_policy |= PCE_FORCE_CONTIGUOUS;

/* How many cluster in 16 Gigabytes */ cluster_size_bytes = pc_cluster_size(VideoBufferDrive); Options.min_clusters_per_allocation = 16 * (1073741823/cluster_size_bytes);

VideoBufferFd = pc_cfilio_open ((byte *)VideoBufferFileName, (PO_BINARY|PO_RDWR| PO_CREAT|PO_TRUNC), &Options);

DisplayFromFd = VideoBufferedFd;

if(VideoBufferedFd >= 0) // Return the Status return(TRUE); else return(FALSE);

}

Buffering Incoming VideoBuffering Incoming Video

Background thread cycles continuously, waitsfor a Framebuffer to be captured then writes tothe circular file and releases the Framebuffermemory for future capture

Buffering Incoming VideoBuffering Incoming Videovoid VideoInputThread(void){

int FrameSize;dword FrameAddress, nWritten;

for(;;){

WaitForFrame(&FrameSize, &FrameAddress);pc_cfilio_write(VideoBufferFd, (byte*)FrameAddress,

FrameSize, &nWritten);ReleaseFrameMemory(FrameAddress, FrameSize);

}}

Displaying Buffered VideoDisplaying Buffered Video

Background thread cycles continuously,acquires an available video display page, thenreads data from the circular file into the videomemory and alerts the video processor that theframe may now be displayed

Displaying Buffered VideoDisplaying Buffered Videovoid DisplayFromVideoBuffer(void){

for(;;){ VideoPage = GetNextFreeVideoPage(&PageAddress, &PageSize); ClaimVideoProcessor();

// Display from the circular file unless we are displaying a saved video clip if (DisplayFromFd == VideoBufferedFd)

pc_cfilio_read (VideoBufferFd, PageAddress, PageSize, &BytesRead); else

{ // Displaying an archive pc_efilio_read(DisplayFromFd, PageAddress, PageSize, &BytesRead);

if (!BytesRead) { // The clip finished. resume displaying from the video buffer

pc_efilio_close(DisplayFromFd); DisplayFromFd = VideoBufferedFd;

pc_cfilio_read (VideoBufferFd, PageAddress, PageSize, &BytesRead); } }

QueueDisplayVideoPage (VideoPage); ReleaseVideoProcessor ();}

}

Accessing Frames In The BufferAccessing Frames In The Buffer

Seek to a specific Frame in the video bufferwhere DisplayFromVideoBuffer() will resumedisplaying data from.

Accessing Frames In The BufferAccessing Frames In The BufferBOOLEAN SetVideoBufferDisplayLocation(dword FrameNumber){ ddword stream_offset; dword CurrLPtrHi, CurrLPtrLo, newFramePtrHi, newFramePtrLo;

ClaimVideoProcessor();// save the current display location

if (DisplayFromFd == VideoBufferedFd) pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, 0, 0,

PSEEK_CUR, &CurrLPtrHi, &CurrLPtrLo); else

pc_efilio_lseek (DisplayFromFd, 0, 0, PSEEK_CUR, &CurrLPtrHi, &CurrLPtrLo);

// Now seek to the byte location of the framestream_offset = FrameNumber * BYTES_PER_FRAME;

if (DisplayFromFd == VideoBufferedFd) SeekOk = pc_cstreamio_lseek (VideoBufferFd, CFREAD_POINTER,

M64HIGHDW(stream_offset), M64LOWDW(stream_offset), PSEEK_SET, &newFramePtrHi, &newFramePtrLo);

else SeekOk = pc_efilio_lseek (DisplayFromFd, M64HIGHDW(stream_offset),

M64LOWDW(stream_offset), PSEEK_SET, &newFramePtrHi, &newFramePtrLo);

Accessing Frames In The BufferAccessing Frames In The Buffer if (SeekOk) { // The read pointer has been moved to this frame allow the display process // to resume from here

ReleaseVideoProcessor(); return(TRUE);

} else {

// Seek failed make sure the pointer is at the original location if (DisplayFromFd == VideoBufferedFd)

// This frame number is not buffered in the 16 gigabyte circular pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, &CurrLPtrHi,

&CurrLPtrLo, PSEEK_SET, &CurrLPtrHi, &CurrLPtrLo); else

// This frame number is not buffered in the video clip pc_efilio_lseek (DisplayFromFd, &CurrLPtrHi, &CurrLPtrLo, PSEEK_SET,

&CurrLPtrHi, &CurrLPtrLo);ReleaseVideoProcessor();return(FALSE);

}}

Buffering At Steady StateBuffering At Steady State

Demonstrate using circular files for a burstdata collection application with deep bufferingrequirements.

Data retention is critical so if the applicationloses any data it is a critical error and must berestarted with more buffering, faster processing or slower collection rates.

Buffering At Steady StateBuffering At Steady Statevoid CollectBurstInput(){ EFILEOPTIONS Options; dword cluster_size_bytes,nWritten,nRead,nBurstdata; int CFileFd; byte *PburstData, Buffer[4096]; rtfs_memset(&Options, 0, sizeof(Options)); /* Create a circular file that wraps at 16Gigabytes and stops writing if the write

pointer overtakes the read pointer */ Options.allocation_policy = PCE_CIRCULAR_FILE; /* Since the file is over 4 gigabytes, Use FAT64 */ Options.allocation_policy |= PCE_64BIT_META_FILE; Options.circular_file_size_hi = 4; /* 4 X 4 gigabytes */ Options.circular_file_size_lo = 0; /* Wrap point */ /* Force it to be contiguous and preallocate all clusters*/ Options.allocation_policy |= PCE_FORCE_CONTIGUOUS; /* Make it a temp file since there's no need to ever flush the file's metadata to disk Options.allocation_policy != PCE_TEMP_FILE; /* How many cluster in 16 Gigabytes */ cluster_size_bytes = pc_cluster_size(DataBufferDrive); Options.min_clusters_per_allocation = 16 * (1073741823/cluster_size_bytes); CFileFd = pc_cfilio_open ("DataCollectBufferFIle",

PO_BINARY|PO_RDWR|PO_CREAT|PO_TRUNC, &Options);if(CFileFd < 0)

return;

Buffering At Steady StateBuffering At Steady State/*Enter a loop, accepting variable amount of data from a fictitious data collection routine and put the data in a ring buffer.

Read the data in 4096 byte chunks from the ring buffer and pass it to a fictitious processing routine. If the processing loop empties the buffer, wait for more data.

If the write operation can't write all of the data that the data collection routine provides that indicates it would overwrite data that has not yet been collected, so warn the user and halt the process*/for (;;)

{ if (BurstDataAvailable(&PburstData, &nBurstdata) {

pc_cfilio_write(CFileFd, PburstData, nBurstdata, &nWritten); if (nWritten != nBurstdata) { printf("File buffer is too small, lost data\n"); pc_cfilio_close(CFileFd); return; } }

pc_cfilio_read(CFileFd, Buffer, 4096, &nRead);

Buffering At Steady StateBuffering At Steady State if (nRead) ProcessData(Buffer,nRead);

else WaitForMoreData(); }}

Archiving Buffered VideoArchiving Buffered Video

Extract some number of frames from the videobuffer to an archive file. The extracted data willbe stored in a linear file, but it will also be partof the video buffer file.

Archiving Buffered VideoArchiving Buffered Video/* Callback function that closes the archive file when it is no longer needed by the

circular file. This occurs when the write pointer reaches and overtakes the region in the buffer where the data was extracted from. */

void CBArchiveOverwritten(int fd, BOOLEAN Success){ //Start an asynchronous close, the background async handler will complete it for us

pc_efilio_async_close(fd);}

BOOLEAN ArchiveFromVideoBuffer(char *ArchiveFileName, dword StartFrame, dword nFrames, BOOLEAN DoFlush)

{ int ExtractFd; ddword stream_offset,extract_bytes; EFILEOPTIONS ArchiveFileOptions; BOOLEAN Result = FALSE;

/* open a special REMAP file for archiving */ rtfs_memset(&ArchiveFileOptions, 0, sizeof(ArchiveFileOptions)); ArchiveFileOptions.allocation_policy = PCE_REMAP_FILE; ArchiveFileOptions.allocation_policy |= PCE_64BIT_META_FILE;

ExtractFd = pc_efilio_open(ArchiveFileName, PO_RDWR|PO_TRUNC|PO_CREAT, PS_IWRITE | PS_IREAD, &ArchiveFileOptions);

if (ExtractFd < 0) return(FALSE);

Archiving Buffered VideoArchiving Buffered Video // This moves the read pointer but happens very quickly, so wait for retrace

WaitForVideoRetrace();ClaimVideoProcessor();//save current display locationpc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, 0, 0, PSEEK_CUR, &CurrLPtrHi, &CurrLPtrLo);

// Now seek to the byte location of the frames to archivestream_offset = StartFrame * BYTES_PER_FRAME;if(pc_cstreamio_lseek (VideoBufferFd, CFREAD_POINTER,

M64HIGHDW(stream_offset), M64LOWDW(stream_offset), PSEEK_SET, &newFramePtrHi, &newFramePtrLo))

{extract_bytes = nFrames * BYTES_PER_FRAME;

/* link the two files together with little or no copying */Result = pc_cfilio_extract(VideoBufferFd, ExtractFd,

M64HIGHDW(extract_bytes), M64HIGHDW(extract_bytes));

}

Archiving Buffered VideoArchiving Buffered Video // restore the pointer to the original location and resume display

pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, &CurrLPtrHi, &CurrLPtrLo, PSEEK_CUR, &newFramePtrHi, &newFramePtrLo);

ReleaseVideoProcessor();

// If success, start an async flush process to put it to disk if (Result) pc_efilio_async_flush(ExtractFd); else { // Otherwise delete the empty extract file pc_efilio_close(ExtractFd); pc_efilio_async_delete(ArchiveFileName); }

return (Result);}

Displaying Archived VideoDisplaying Archived Video

Open a saved video clip for display by thedisplay thread.

Displaying Archived VideoDisplaying Archived Videoint CurrentDisplayngArchiveFd;void DisplayFromVideoArchive(char *ArchiveFileName){

int ArchiveFd; EFILEOPTIONS ArchiveFileOptions;

// Reopen the archive file, the file will automatically detect FAT64 or FAT32 rtfs_memset(&ArchiveFileOptions, 0, sizeof(ArchiveFileOptions));

ArchiveFd = pc_efilio_open(ArchiveFileName, PO_RDWR,0, &ArchiveFileOptions); if (ArchiveFd >= 0) { // Set the file descriptor for the display task ClaimVideoProcessor(); DisplayFromFd = ArchiveFd; ReleaseVideoProcessor(); }}

Completing Asynchronous Completing Asynchronous RequestsRequests

This background thread cycles continuously andwaits for a time window when it can safelyaccess the drive without affecting foregroundoperations.It then performs one or more cycles of theasynchronous processing.

Completing Completing AsynchronousAsynchronous Requests Requestsvoid AsynchronousCompletionThread(void){

int AsyncState, StepPerPass, DriveNumber;dword FrameAddress, nWritten;

DriveNumber = GetVideoDriveNumber();

for (;;) { // Steal cycles for asynchronous processing during a retrace WaitForVideoRetrace();

StepsPerPass = 1;

/* Do asynchronous processing, on Files, buffers, journal flushes and journal to

volume restores. We could also choose to process only up to an earlier state, such as Journal flush completion by specifying DRV_ASYNC_DONE_JOURNALFLUSH. We could then in a less time critical period complete the whole process. */

AsyncState = pc_async_continue(DriveNumber, DRV_ASYNC_DONE_RESTORE, 1);

Completing Completing AsynchronousAsynchronous Requests Requests if (AsyncState == PC_ASYNC_COMPLETE)

{// All done, set a fictitious VolumeUptoDate variable needs to be done

VolumeUptoDate = TRUE; } else if (AsyncState = PC_ASYNC_CONTINUE)

{ /* We know that we'll only perform on single multi-block read or one single

multi-block write, and we know we can do this in less than 10 miliseconds, so if we have more than 10 miliseconds left take another pass */

if (TimeLeftInRetrace() > 10) AsyncState = pc_async_continue (DriveNumber,

DRV_ASYNC_DONE_RESTORE,1); } if (AsyncState = PC_ASYNC_ERROR) { printf ("Error in processing, halt...\n"); halt (); } }}

RTFSProPlus PortingRTFSProPlus Porting

Ports - Ports - LinuxLinux VxWorksVxWorks

SignalingSignaling pthread_mutex_init()pthread_mutex_init()

pthread_mutex_lock()pthread_mutex_lock()

pthread_mutex_unlock()pthread_mutex_unlock()

semBCreate()semBCreate()

semTake()semTake()

semDelete()semDelete()

TimingTiming usleep()usleep()

gettimeofday()gettimeofday()sleep()sleep()tickGet()tickGet()

KernelKernel pthread_self()pthread_self() taskIdSelf()taskIdSelf()

RTFSProPlus RAM FootprintRTFSProPlus RAM Footprint

Journal Restore Buffer 32K up to no limit

Journal Block Remap Buffer 32K is essentially infinite

Cluster Management Buffer 32K up to no limit

Fat table Buffer 1K is min, 10K is practical

Fragment Buffer 12 bytes/fragment, 120K is essentially infinite

FAT32 Extended Files 200 bytes per file

FAT64 Metafiles 3200 bytes per file

FAT64 Circular Files 4000 bytes per file

RTFSProPlus Code SizeRTFSProPlus Code Size

220K (approx) with all features enabled, non-optimized, full debug support

implementing a fat based 64 bit dvr

Documents