Firmware compilation using GCC

[afanofosc]

Recreating (portions of) a recent topic from the old forums.

Hi,

I am working on compiling the firmware with GCC in order to make it easy for anybody to hack the original firmware.

I would like to make it easier for any curious user to work on the firmware using GCC or IAR, to share its work, and to track changes made to the original firmware. For this, I am building a site which will contain instructions on how to build the firmware, which will try to explain firmware internals, and which provides a decentralised source repository.

I started this project after some frustration about:

• NXTGCC:
  • not stable enough,
    distributed with a full Windows toolchain and Eclipse installation for which I have no usage,
    no source repository, poor visibility on what changed and why,
    no IAR support for those who like it.
  NXC Enhanced firmware:
  • can not compile using GCC,
    no source repository, bad visibility on what changed and why,
    no release versions, so its hard to know what is exactly in a binary.
    other firmware:
    not compatible with NXC which I like to use.

I greatly appreciate work by John Hansen and Rasmus Ulslev Pedersen, I just want more


I am going on vacation Saturday until 2010-08-08, so this website will not be ready before I leave, but you can try some firmware yet. I think I have solved problems of NXTGCC (you will find how in the related commits).

So if you want to try it, download http://ni.fr.eu.org/~nico/nxt_firmware.rfw to your brick and tell me about any problem you can see. Warning: this will erase all the files in your user file system. This is a stock 1.29 firmware.

I you would like to test GCC compilation before the website is ready, please let me known.

hi,
sounds cool!
can you please tell what's it all about with this fw?
is there a home page we can read sth about your fw development?
what's different?
what's better?
what's easier?
which IDE to use?
what syntax - c or c++ or NXC?
support of dynamic memory, pointers, recursion?
what API?
do you have program examples maybe using multithreading (bumper car, line follower, maze solver)?
informations greatly appreciated!

hi,
sounds cool!
can you please tell what's it all about with this fw?
is there a home page we can read sth about your fw development?

Not yet, but I am working on it. It should be ready in three weeks.

what's different?
what's better?
what's easier?
which IDE to use?
what syntax - c or c++ or NXC?
support of dynamic memory, pointers, recursion?
what API?
do you have program examples maybe using multithreading (bumper car, line follower, maze solver)?
informations greatly appreciated!

For the moment, it should be exactly the same as the original LEGO firmware. The goal is to permit people to make change easily to the original firmware. There is no benefit yet for the end user. There is only benefit for people wanting to experiment with modifying the firmware.

I am leaving now, see you in two weeks

Some of my "enhancements" involve refactoring and changes to reduce the code and data size of the resulting build. Are you willing to have those kind of check-ins or would you prefer to limit changes to specific feature additions and bug fixes?

One change, for example, involved making the compiler avoid as much as possible all float to double or vice versa conversions that were showing up in the IAR map file. So floating point constants were declared with a trailing F or prepended with (float) in a few places. Where the 1.28 source code had calls to sin, cos, tan, etc... (removed from the compiled code via #if 0) I used the sinf, cosf, tanf, etc... functions instead. Another example is that I removed duplicate strings from the UI.txt file and rearranged the constants used in Functions.inl which referred to those duplicate strings. Repetitive code in c_input.c within a large case statement has been extensively refactored as well. None of these are necessarily enhancements or bug fixes and it makes it harder for me to do comparisons back to the original LEGO code but I would prefer to convince LEGO to switch to my implementation rather than consider rolling back those kind of changes in my code.

John Hansen

... but please no reduction for double to float but enhancement for double instead if possible )

No you really need an EABI toolchain.

This is explained here: http://nxt-firmware.ni.fr.eu.org/build/gcc/

Hmm. I'm still confused regarding the EABI vs. ELF toolchains. Googling doesn't reveal much info.
Prior to this I was building NxOS successfully using the Yagarto ELF toolchain on Mac OSX, and it seems to run fine.
According to ARM Info Center, they've adopted EABI since ~2003 and the BSABI document (ARM IHI 0036B, current through ABI release 2.08 10th October 2008, reissued 28th October 2009) does not mention EABI explicitly.

From what I understand, ELF vs. EABI toolchains would probably affect ARM platforms which has some OS functionality to support shared libraries, etc. Since the NXT Firmware is standalone (and the linker map definition files have to be custom generated), I'm not sure what is the impact?

From what I understand, ELF vs. EABI toolchains would probably affect ARM platforms which has some OS functionality to support shared libraries, etc. Since the NXT Firmware is standalone (and the linker map definition files have to be custom generated), I'm not sure what is the impact?

Replying to myself....
I tried compiling NxOS (this is the only point of reference I have, I couldn't compile the NXT firmware using arm-eabi-gcc from devKitPro for Mac OS X due to some issues with their pre-built newlib) using arm-eabi-gcc vs. arm-elf-gcc, and it gave the following file sizes:
57675 24 Aug 13:41 tests_rxe.elf_arm-eabi-gcc-4.5.0
59637 24 Aug 13:39 tests_rxe.elf_arm-elf-gcc-4.5.0

The main impact (code size reduction) comes from arm-eabi-gcc generating Thumb instructions, whereas arm-elf-gcc defaults to ARM instructions.

A useful (?) comparison of ARM EABI vs. older (non-EABI) formats is available here

Some of my "enhancements" involve refactoring and changes to reduce the code and data size of the resulting build. Are you willing to have those kind of check-ins or would you prefer to limit changes to specific feature additions and bug fixes?
...

This definitely worth the change, I also find that the use of double and float are not consistent.

Moreover, using git will make merging from LEGO easier.

Hmm. I'm still confused regarding the EABI vs. ELF toolchains. Googling doesn't reveal much info.
Prior to this I was building NxOS successfully using the Yagarto ELF toolchain on Mac OSX, and it seems to run fine.
According to ARM Info Center, they've adopted EABI since ~2003 and the BSABI document (ARM IHI 0036B, current through ABI release 2.08 10th October 2008, reissued 28th October 2009) does not mention EABI explicitly.

From what I understand, ELF vs. EABI toolchains would probably affect ARM platforms which has some OS functionality to support shared libraries, etc. Since the NXT Firmware is standalone (and the linker map definition files have to be custom generated), I'm not sure what is the impact?

Quoting http://wiki.debian.org/ArmEabiPort:

With the new ABI, default structure packing changes, as do some default data sizes and alignment (which also have a knock-on effect on structure packing). In particular the minimum size and alignment of a structure was 4 bytes. Under the EABI there is no minimum and the alignment is determined by the types of the components it contains. This will break programs that know too much about the way structures are packed and can break code that writes binary files by dumping and reading structures.

That is the problem with the APCS toolchain (arm-elf-gcc). The NXT firmware wrongly use knowledge about the compiler way of assembling structures. The new EABI match the convention used in the IAR compiler.

Given:

Code: Select all

struct a {
    short int s;
};

IAR and EABI toolchain will make a 2 byte structure, APCS will make a 4 byte structure. The NXT source code wants the IAR convention.

I will add this to the FAQ section to make this clearer.

Also, arm-elf is being deprecated: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42100

Hello
Just an idea here, 1/3 of the weight of the firmware source code is about the "try me" programs, in my thought, not needed in "optimized" firmware, even in common use. this is an easy improvement, without touching any compatibility NXT-G/NXC. Am i wrong, or?
Best regards
ben

P.S Nicolas, belle initiative que voilà! j'habite Cachan, tu passes souvent sur la région parisienne?

The programs themselves are not part of the source code - just part of the firmware image as files that you can run. I'm sure neither my IAR-built firmware nor this one includes those files by default. There is a chunk of firmware code that is all about writing on-brick programs. That could be stripped out without breaking the VM. Also the on-board datalogging support could be stripped out as could the manual sensor value inspection features of the user interface. But I don't think that would make a very big difference in code size. Most of the code is in the VM and the various firmware modules that interface with outputs, inputs, buttons, comm of three sorts, i2c support, etc... None of that could be removed without breaking compatibility with NXT-G and NXC.

John Hansen

What about the graphics used in those menus? When I skimmed though the source it looked like there was used quite a bit of graphics, but I don't know how much space this actually ends up taking.
The small animation when you turn on the NXT, does it actually do anything in the background? Because I have always found it annoying to wait for it to finish.

Anyway, I would have nothing against removing those from the menu, not because of the file size, but because it is annoying to have random stuff in the menu you will never use anyway.

The graphics would probably only affect the size of the data segment and not the code segment but removing them would reduce the total firmware image size. At least that is how things work in IAR. Since the LEGO MINDSTORMS edition of the IAR compiler is code segment size limited that's the only bit that I have been paying close attention to. It would be a lot of work to strip out all the bits that aren't critical to running code on the VM. But it might be worth creating a version of the firmware that is "streamlined" in that way.

John Hansen

Hello,

What about the graphics used in those menus? When I skimmed though the source it looked like there was used quite a bit of graphics, but I don't know how much space this actually ends up taking.

Looking at the object files there are two big components: The virtual machine and the user interface. So I think by optimizing the user interface the firmware could be shrinked significant. But that depends on the needs of the people.

Perhaps I will try a little bit around but I cannot promise anything...

Bye marvin

Removing the start-up animation should save 4k just for the bitmaps, so this is a good candidate. I also find the start-up time too long and the brick is not doing anything useful during this interval.

About removing menus, I am not sure if this can easily be done without changing everything. I am not familiar with the menu system yet.

You can evaluate the weight of a feature by looking at the sym file (make sym).

Anyway, I do not want to drop everything, but I could make a configuration system to choose at build time. I could also build binaries with several "flavor" (light, medium, fat...).

Do you feel you need more user space?

need more user space?
you can't ever have enough of it!

Hello,

Removing the start-up animation should save 4k just for the bitmaps, so this is a good candidate.

I tried yesterday and it saves about 2 k. I think the bitmaps are stored more efficient in the firmware.

I also find the start-up time too long and the brick is not doing anything useful during this interval.

That can be easily changed in the c_ui.c code. Or maybe in c_ui.h.

About removing menus, I am not sure if this can easily be done without changing everything. I am not familiar with the menu system yet.

I tried yesterday and I got a slim menu version compiled. But I do not have a simulator and the real brick is at home - not here. So I cannot try the result.

The memory savings is not so much - only about 2 k again. When removing all the related functions and all of the bitmap icons this will increase the memory savings. But I am not sure about the sense of this. It will be a lot of work to get perhaps 10 k of more free memory. This will be bug sensitive - the work has to be done very concentrated.

I think for most of the people the free memory in the brick is enough. For the others 10 k more free memory is even not enough so I suggest to leave this as it is.

Bye marvin

I tried yesterday and it saves about 2 k. I think the bitmaps are stored more efficient in the firmware.

I got my number by looking at the compiler output, I am really surprised you did not save more space. Could you publish your patch?

I think my opinion on this is pretty much in agreement with marvin's. Unless we see a substantial increase in available user flash memory it's probably not worth the time and effort.

John Hansen

The cache does not include the more recent posts from this thread (or all of page 1).

[schodet]

Thanks for importing this.

There was the first release without the introduction animation:

http://nxt-firmware.ni.fr.eu.org/binaries/

I am preparing an improvement about absolute position regulation, I hope I can publish a first version in the next few days.

[afanofosc]

I made a change to the enhanced NBC/NXC firmware which enabled a hold at position mode which would (in theory) cause the firmware to hold the motor at the specified tachometer target when it was reached.

Code: Select all

#if defined(__ENHANCED_FIRMWARE) && (__FIRMWARE_VERSION > 107)
/** Options field. Contains a combination of the output options constants. Read/write.
 *  Set options for how the output module will act when a tachometer limit is reached. Option
 *  constants can be combined with bitwise OR.  Use OUT_OPTION_HOLDATLIMIT to have the output
 *  module hold the motor when it reaches the tachometer limit.  Use OUT_OPTION_RAMPDOWNTOLIMIT
 *  to have the output module ramp down the motor power as it approaches the tachometer limit
 *  (not yet implemented). */
#define OutputOptions   15
#endif


/** @defgroup OutOptionConstants Output port option constants
 * Use these constants to configure the desired options for the
 * specified motor(s): hold at limit and ramp down to limit. Option constants
 * can be combined with bitwise OR.
 * \sa SetOutput()
 * @{
 */
#if defined(__ENHANCED_FIRMWARE) && (__FIRMWARE_VERSION > 107)
#define OUT_OPTION_HOLDATLIMIT     0x10 /*!< Option to have the firmware hold the motor when it reaches the tachometer limit */
#define OUT_OPTION_RAMPDOWNTOLIMIT 0x20 /*!< Option to have the firmware rampdown the motor power as it approaches the tachometer limit (not implemented yet) */
#endif
/** @} */  // end of OutOptionConstants group

I haven't implemented the ramp down to limit option at all but I think the hold at limit might work. It's been ages since I made those changes and didn't do a lot of testing so it may not work correctly.

John Hansen

[schodet]

I made a change to the enhanced NBC/NXC firmware which enabled a hold at position mode which would (in theory) cause the firmware to hold the motor at the specified tachometer target when it was reached.

I was about to add new output fields, we better have to synchronise for such changes.

My change introduces a new regulation mode with more precise position control and the ability to change the consign at any time. This can not be done with the current LEGO algorithm. I use this kind of control on bigger robots (you can see it in action here for example: http://apbteam.org/wiki/Eurobot/2010/Report).

[afanofosc]

Mindsensors has a firmware patch on their website. Here's the contents:

Code: Select all

--- orig/d_lowspeed.r	2010-05-19 16:00:22.421657400 -0400
+++ d_lowspeed.r	2010-05-19 15:54:10.220657400 -0400
@@ -118,7 +118,7 @@
 										  *AT91C_PWMC_MR	   |= 0x600;					/* Prescaler MCK divided with 64 */\
 										  *AT91C_PWMC_CH0_CMR   = 0x06;						/* Channel 0 uses MCK divided by 64 */\
 										  *AT91C_PWMC_CH0_CMR  &= 0xFFFFFEFF;				/* Left alignment on periode */\
-										  *AT91C_PWMC_CH0_CPRDR = 0x20;						/* Set to 39 => 52uSecondes interrupt */\
+										  *AT91C_PWMC_CH0_CPRDR = 0x1C;						/* mindsensors: - changed to 1c Set to 39 => 52uSecondes interrupt */\
 										  *AT91C_PWMC_IDR	    = AT91C_PWMC_CHID0;			/* Disable interrupt for PWM output channel 0 */\
 										  *AT91C_AIC_IDCR       = 0x400;					/* Disable AIC intterupt on ID10 PWM */\
                                            AT91C_AIC_SVR[10]    = (unsigned int)LowSpeedPwmIrqHandler;\
@@ -476,6 +476,7 @@
             case TX_DATA_CLK_HIGH:
             {
               SETClkLow(ChannelNr);
+						  int a; for (a=0; a<60; a++);  // mindsensors:
               if (LowSpeedData[ChannelNr].MaskBit == 0)     				                           //Is Byte Done, then we need a ack from receiver
             {
               SETDataToInput(ChannelNr);                                                             //Set datapin to input

Can anyone comment as to whether and why this would be a good change to make to the firmware? I don't really understand what they are trying to improve.

John Hansen

[tcwan]

Mindsensors has a firmware patch on their website. Here's the contents:

Code: Select all

             case TX_DATA_CLK_HIGH:
             {
               SETClkLow(ChannelNr);
+						  int a; for (a=0; a<60; a++);  // mindsensors:
               if (LowSpeedData[ChannelNr].MaskBit == 0)     				                           //Is Byte Done, then we need a ack from receiver
             {
               SETDataToInput(ChannelNr);                                                             //Set datapin to input

Can anyone comment as to whether and why this would be a good change to make to the firmware? I don't really understand what they are trying to improve.

John Hansen

From just looking at the code, it is a busy loop delay, which happens after toggling the Clock line. I presume that is to let the logic settle before checking the line. Maybe there is sufficient loading on the signal lines that it won't settle in time?

The danger of doing it as a busy loop is that Optimizing compilers may detect that it is not doing anything useful and optimize it out!

[gloomyandy]

Hi,
Yes the busy loop is to let the i2c lines settle. It turns out that with the original code when using a sensor mux and long cables, some sensors saw an i2c reset because the clock had not dropped to zero before the data line was changed. Hence the small delay. I've added a similar change into the leJOS firmware and can confirm both the problem and that the leJOS fix works... I agree about the problem with optimizing compilers, but it is tricky to arrange these kinds of very short delays by other means...

Andy

[gloomyandy]

Oh and I think the other change is basically making a slight increase in the speed of the clock being used for i2c... I did not add this to leJOS...

Andy

[afanofosc]

Would changing the clock slightly benefit some i2c devices or could it possibly cause problems for some? Or does it just slightly speed up the rate at which data is sent back and forth between the NXT and the device?

John Hansen

[gloomyandy]

Hi John,
To be honest I'm not really sure. Obviously making it faster will result in faster transfers (though in this case the change is pretty small I think). But that change may screw up some devices. I suspect that any hardware based i2c device will work fine, it is the bitbanged devices that may have problems... The most obvious one is the Lego ultrasonic sensor. That has all sorts of strange timing requirements. I decided to play it safe and stick to the original Lego clock speed. I seem to remember that RobotC offers normal speed (with full Lego compatibility), plus various faster modes (and when you use the faster modes the Lego ultrasonic sensor does not work). At some point I may add something similar to leJOS...

Andy