What are linear files in NXC?

[markcrosbie]

Hi all,

I've searched the NXC documentation and LEGO firmware spec documents, and I can't find anything that explains the difference between a linear and non-linear file. I see that there are parallel open/read/write/close functions for both file types, but that is the difference? When should a linear file be used in preference to a non-linear file?

Thanks,
Mark

[afanofosc]

Mark,

My apologies for not replying already to your PM on this subject. Certain types of files are required by the firmware to be linear which is to say contiguous sectors in the flash memory. Non-linear or data files can be fragmented in the file system. Icon (ric) and executable (rxe, rtm, and sys) files are required to be linear. So the only real reason to use the linear version of the open/create API functions would be to generate an icon or executable file within your program. I have not actually experimented with this at all but, in theory, you could use these options to create functioning ric and rxe files on the NXT. A simple test would be to #import into a byte array an RIC file and then write the array to a linear file and then see if you could draw it to the screen using GraphicOut. Or if you were more adventurous you could write a program that allows a user to create a simple RIC file on the NXT via some fancy button/sensor/motor-based UI and then save the results to an RIC file. I've thought fleetingly about trying something like that a number of times in the past.

This seems to work:

Code: Select all

// letters.ric
desc(0, 96, 8);
sprite(1, 
  0xCC3F0FC3E0783E0F8208781E, 
  0xCC3F0FC3F0FC3F0FC318FC3F, 
  0xCC0C030330CC330CC318CC30, 
  0xFC0C030330CC330CC1B0CC30, 
  0xFC0C0303F0CC330FC1B0FC30, 
  0xCC0C0303E0CC330F80E0FC30, 
  0xCC3F030300FC3F0DC0E0CC3F, 
  0xCC3F030300783E0CC040CC1E);
copybits(0, 1, arg(0), 0, 10, 8, 0, 0);

Compile the above to letters.ric so that it can be downloaded and used in the #import in the following NXC program:

Code: Select all

#download "letters.ric"
#import "letters.ric" data

int Values[] = {0};

void Display( string name, int n )
{
  Values[0]=n*10;
  GraphicOutEx( Values[0],0, name, Values, true );
  Wait( 200 );
}

task main()
{
  remove("letters2.ric");
  for( int i=0; i<9; i++ )
    Display( "letters.ric", i );
  Wait(SEC_5);
  ClearScreen();

  Wait(SEC_2);
  unsigned int fsize, result;
  byte handle;
  fsize = ArrayLen(data);
//  result = CreateFile("letters2.ric", fsize, handle); // doesn't work
  result = CreateFileLinear("letters2.ric", fsize, handle); // works
  if (result == LDR_SUCCESS)
  {
    result = Write(handle, data);
    fclose(handle);
    for( int i=0; i<9; i++ )
      Display( "letters2.ric", i );
    Wait(SEC_5);
  }
}

John Hansen

[markcrosbie]

Thanks John, I suspected it was something like that. I was wondering if linear files had any performance advantage over "regular" files, or indeed if they had a performance disadvantage. I suppose some experiments are in order!

Mark

[HaWe]

where did you first read this misleading word "linear" or - respectively - who introduced this word for this property?
IMHO I would suggest to use "unfragmented" or "coherent" instead.

[markcrosbie]

Hi Doc,

I read the "linear" name in the NXC API documentation, in the NXCDefs.h file and in the official LEGO Executable File specification document. Hence my curiousity.It is an official LEGO term, not one I or John made up. Linear is a good choice of word here - the FLASH disk sectors have to be arranged in linear contiguous order in the filesystem to work. I suspect there is some memory-mapping going on under the hood to map the linear file contents directly into the address space of the ARM processor, much like the UNIX mmap() system call does. If a file was in random sectors in the FLASH filesystem there would be no way to map it into the address space and hence "execute" the contents of the file.

As for the performance impact, I knocked out this quick-n-dirty test harness to see the difference in read/write performance. It would seem that linear files are significantly slower (898ms vs 537ms on my NXT) than regular files. Granted this is a rough test and a single data-point, but good to know otherwise.

Btw John - unless I've made a big error in my code (always possible) the NeXTTool on my Mac does not display the linear file in the file browser, only the regular file. Is that expected behaviour?

Mark

Code: Select all

//
// Test performance of filesystem operations
//
// Test 1 - Is writing/reading a linear file faster than
// a regular file in NXC?

#define FILENAMELINEAR "linear.txt"
#define FILENAMEREGULAR "regular.txt"
#define FILESIZE 10240
#define ITERS 1000

task main() {

	int f;
	int size, r, count, i;
	unsigned int s_l, e_l, s_r, e_r;
	
	byte data[FILESIZE];
	
	ArrayInit(data, 123, FILESIZE);	
	
	r = CreateFileLinear(FILENAMELINEAR, FILESIZE, f);
	
	switch(r) {
	 case LDR_FILEEXISTS:
		DeleteFile(FILENAMELINEAR);
		r = CreateFileLinear(FILENAMELINEAR, FILESIZE, f);
		break;
		
		case LDR_SUCCESS:
		break;
		
		default:
		TextOut(0, LCD_LINE1, "Create Linear fail");
		Wait(2000);
		StopAllTasks();
	}
	
	count = FILESIZE;
	s_l = CurrentTick();
	for(i=0; i < ITERS; i++) {
		r = WriteBytes(f, data, count);
		r = ReadBytes(f, count, data);
		r = fseek(f, 0, SEEK_SET);
	}
	e_l = CurrentTick();
	
	CloseFile(f);
	
	////////// Regular file
	
	r = CreateFile(FILENAMEREGULAR, FILESIZE, f);
	
	if(r == LDR_FILEEXISTS) {
		DeleteFile(FILENAMEREGULAR);
		r = CreateFile(FILENAMEREGULAR, FILESIZE, f);
	}
	
	count = FILESIZE;
	s_r = CurrentTick();
	for(i=0; i < ITERS; i++) {
		r = WriteBytes(f, data, count);
		r = ReadBytes(f, count, data);
		r = fseek(f, 0, SEEK_SET);
	}
	e_r = CurrentTick();
	
	CloseFile(f);

	TextOut(0, LCD_LINE1, "Linear Test");
	TextOut(0, LCD_LINE2, FormatNum("%d ms", e_l - s_l));

	TextOut(0, LCD_LINE4, "NonLinear Test");
	TextOut(0, LCD_LINE5, FormatNum("%d ms", e_r - s_r));
	
	Wait(5000);
}

[afanofosc]

The NXT Explorer tool in NeXT Tools for Mac OSX should show files regardless of whether they are linear or not - to the best of my knowledge, anyway. It shows (or certainly should show) RXE and RIC files which are linear but maybe something weird is going on when you create a linear file with an unexpected file extension. If you power off and power back on the NXT does it change whether the linear file shows up or not? I have seen cases where an improperly closed handle can foul up the find first/find next file iteration. If you see a difference after a power cycle then something must be leaking a handle or something as it definitely looks like you are properly closing your files. Can you tell whether the file contents are what you expect?

I'm glad to see someone using the file seek ability that I added to the enhanced firmware. Have you used it in other programs? Does it appear to work correctly in all the cases where you have used it?

John Hansen

[markcrosbie]

Hi John,

I think the files are being closed correct, so it must be something to do with non-standard extensions for linear files. An interesting corner-case but not something serious.

As for fseek/ftell - well this will give you a laugh: I was writing some file I/O tutorials and grumbling that there was no C-lib file functions when I discovered that I was actually working from an NXC Guide that was over a year old and the firmware supported the file pointer functions all along! Doh! So yes I will be testing them to verify operation, and also testing them when used with ResizeFile (and even linear files

A question though - we now seem to have two branches of file functions in NXC; the ever-growing list of C-library functions (e.g. fseek, ftell, rewind, fopen, fprintf etc) and the NXC "classic" functions such as OpenFileRead and OpenFileAppend. Which will be the set that people should migrate to over time? If I'm writing for an audience a year or two in the future I'd prefer to tell them that NXC supports standard C library functions and that they should use those. What's your thoughts here.

Regards,
Mark

[HaWe]

If I may dare to say - in my humble opinion it should be the ANSI C like commands in any case (the rest optional) to make NXC ACAPLC (as close as possible like C)

ps
what's the opposite to linear?
bent?

[muntoo]

If I may dare to say - in my humble opinion it should be the ANSI C like commands in any case (the rest optional) to make NXC ACAPLC (as close as possible like C)

You've just coined a new term. (I think it should be: ACDC - Almost Completely Disputably C.) *

I think that it's best to use OpenFileRead/Write/Append, as those are supported by the firmware. Use the others only if you have to. (For me, that translates to: "only if it's easier" )

* I shudder at the thought of ACDC++

ps
what's the opposite to linear?
bent?

Fragmented.

[HaWe]

no, the opposite to fragmented is "contiguously", not "linear", and so the opposite to contiguously is fragmented

* I shudder at the thought of ACDC++

ACDC++ is really sidesplitting !!