There seems to be nothing on any of the web forums about the inner workings of the ps3fb driver, so I took a look at the source code in the 2.6.21-rc5 kernel.

The driver code ps3fb.c shows the following interesting table :-

static const struct ps3fb_res_table ps3fb_res[] = {
/* res_x,y margin_x,y full */
{ 720, 480, 72, 48 , 0},
{ 720, 576, 72, 58 , 0},
{ 1280, 720, 78, 38 , 0},
{ 1920, 1080, 116, 58 , 0},
/* full mode */
{ 720, 480, 0, 0 , PS3FB_RES_FULL},
{ 720, 576, 0, 0 , PS3FB_RES_FULL},
{ 1280, 720, 0, 0 , PS3FB_RES_FULL},
{ 1920, 1080, 0, 0 , PS3FB_RES_FULL},
/* vesa: normally full mode */
{ 1280, 768, 0, 0 , 0},
{ 1280, 1024, 0, 0 , 0},
{ 1920, 1200, 0, 0 , 0},
{ 0, 0, 0, 0 , 0} };

The margin_x,y figures (78x and 38y pixels in the case of 720P) are the cause of the black border+scaling in 720P without overscan. Note that the fullscreen modes all specify a margin of zero, but have an additional PS3FB_RES_FULL flag to switch on overscan.

Next step is to modify the table so that margin_x,y=0 for non-fullscreen resolutions and recompile the kernel+all its modules. That will take a loooong time to do on a PS3, so I don't expect to report much progress in the immediate future.

If this works then 720P should be improved a lot - no borders, no scaling, no silly overscan.

It also seems to me that there's scope for adding modes to the driver. It will be a lot more complicated than removing the border, but I think it should be possible to hack in an extra mode for 1366x768.

For more of a clue on what's involved in adding a mode, check out the following function in ps3fb.c :-
static const struct fb_videomode ps3fb_modedb[]
I have no idea what half the numbers in this table actually do, so that's going to take some detective work.