[rsbac] Re: Hang on boot with 2.6.5

[Amon Ott]

On Samstag, 17. April 2004 02:19, Rob See wrote:
>     I having a problem booting the 2.6.5 kernel, using rsbac 1.2.3-pre4. 
> Right after init starts, I get the following messages:
> rsbac_get_super_block(): auto-mounting device 00:07
> rsbac_acl_get_single_right(): Could not lookup device!
> rsbac_acl_check_right(): rsbac_acl_get_single_right() returned error 
> RSBAC_EINVALIDDEV! rsbac_adf_request(): request CLOSE, pid 43, ppid 42, 
> prog_name rc, uid 0, target_type FIFO, tid Device 00:07 Inode 2263 Path 
> pipe:/[2263], attr none, value 0, result NOT_GRANTED by GEN ACL

This is an access on pipefs, where unnamed pipes are stored, e.g. when 
using | in bash. It is a mount race and in this case completely harmless - 
CLOSE will always work despite this message.
 
>     I've seen references to similar problems on the mailing list, but 
> not any solutions. This problem does go away if I used 
> rsbac_delayed_init, but on the next boot, the kernel hangs when all of 
> the local filesystems are mounted. At this point, I have to turn on 

The locking around parallel mounts is the most critical part in all SMP 
code in RSBAC. Almost all SMP related hangs happen here, whatever I do to 
get them away. Once the system is up and running, all is fine.

> either rsbac_delayed_root=99:99, or rsbac_softmode to get the machine to 

So it does work with softmode in this case? That's something new, but it 
all depends on timing.

I assume that you are using delayed init, e.g. because of an initrd. Does 
it work without initrd and without delayed init?

> boot. I tried using rsbac_debug_adf_all, but got no more output than 
> with without it. I have not for the life of me been able to figure out 

rsbac_debug_stack will show the steps in mounting, there is some debug code 
to detect stack overflows.

> what device 00:07 is. I've started the kernel with init=/bin/sh, and can 
> trigger this auto-mounting device 00:07 error by doing a simple ls in 

In your case it is the internal pipefs, you can see it from the logged file 
path.

> the current directory.  I have seen rsbac successfully make it through 
> the auto-mounting device 00:07 step before, but I can't figure out what 
> I'd done differently to get it to boot.

The difference is that now there is a /rsbac.dat directory on each of your 
real filesystems, from which attribute data get loaded at mount time. 
These disk accesses can eventually sleep while holding the mount lock. If 
at the same time another parallel mount tries to do the same, it can 
deadlock.

Additionally, there is a spinlock when adding an RSBAC device item, which 
blocks while another process reads attributes. A lot of fun with parallel 
tasks...

What you can do as a test is change your startup script to mount each 
filesystem individually with 1s sleep in between.
 
>     This kernel is being run on a Dell PowerEdge 1750 with 2 Gigs of 
> RAM, and dual 2.4Ghz  Xeons configured for hyperthreading (shows up as 4 
> procs to the kernel)

I have a very similar SMP system here and will run some more test on it 
soon. Could you please send me your .config for these tests?

Amon.
-- 
http://www.rsbac.org - GnuPG: 2048g/5DEAAA30 2002-10-22