robocoder: Thanks for having a look, but I don't think the problem in my iptables target is the search for the "fatso" - that part looks correct to me.
Here's the snippet again. (Note that tasklist_lock does get unlocked in the code that follows this snippet!):
struct task_struct *p, *fatso = NULL;
int fatso_rss;
/* Find the "fatso" process -- the one with the most RSS */
read_lock(&tasklist_lock);
for_each_task(p) {
spin_lock(&p->mm->page_table_lock);
if (fatso == NULL || (p->mm->rss > fatso_rss)) {
fatso = p;
fatso_rss = p->mm->rss;
}
spin_unlock(&p->mm->page_table_lock);
}
The "post condition" is that fatso should always be set to the process with the most RSS, and it looks correct to me:
In the first iteration, fatso is NULL so the if will be entered, causing fatso and fatso_rss to be initialized based on the first process. The uninitialized value of fatso_rss wasn't used because fatso == NULL is true in the condition.
On each other iteration (one for each other process), if that process has more RSS then it will replace fatso. Eventually the process with the most RSS will be considered and will replace the current fatso, and then won't be replaced by another process because none has more RSS. So after all processes are considered the fatso should be the process with the most RSS.
Or am I missing something trivial?
I've been expecting the problem to be some subtle kernel issue, like perhaps it being illegal to take the process list lock from the context where iptables rules are invoked. It's certainly true that suspicions like that tend to blind one to his own embarrassing programming errors though :-). I have tried moving the "real work" into a kernel thread, but that didn't help.
