Thread-affinity API issues under RedHat

[ewmayer]

(Note: The code in question compiles fine under OS X, debian and ubuntu - I've no convenient interactive access to a multicore setup running RH).

Got a compiler-error note from someone trying to compile Mlucas-multithreaded. Here are his OS details:

$cat /proc/version
Linux version 2.6.32-358.18.1.el6.x86_64 (mockbuild@c6b10.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Aug 28 17:19:38 UTC 2013

Here is the error he is getting:

threadpool.c: In function ‘worker_thr_routine’:
threadpool.c:260: warning: implicit declaration of function ‘syscall’
threadpool.c:260: error: ‘__NR_gettid’ undeclared (first use in this function)
threadpool.c:260: error: (Each undeclared identifier is reported only once
threadpool.c:260: error: for each function it appears in.)
threadpool.c:264: warning: implicit declaration of function ‘CPU_ZERO’
threadpool.c:266: warning: implicit declaration of function ‘CPU_SET’
threadpool.c:267: warning: implicit declaration of function ‘sched_setaffinity’
threadpool.c:268: warning: implicit declaration of function ‘CPU_ISSET’

Here is my current include-file code for thread-affinity stuff - locally pasting this into a text-highlighted .h file should make it more readable:

/* Online docs [e.g. http://www.kernel.org/doc/man-pages/online/pages/man2/sched_setaffinity.2.html]
tell us that in order to access macros for `cpu_set', we must #define _GNU_SOURCE before including <sched.h>.
However, whether I define the above or not, on my distro (Fedora v16), I get these compile-time warnings
indicating that the affinity stuff is not being included:

	"warning: implicit declaration of function ‘CPU_ZERO’", etc.

Some sleuthing reveals that (at least in my distro) sched.h #includes <features.h>,
and the latter header then defines __USE_GNU if _GNU_SOURCE is defined.
In other words, #define _GNU_SOURCE simply causes __USE_GNU to also get defined.
Or at least it should, but my above-quoted build diagnostics indicate otherwise.
Even more bizarrely, when (in addition to defining just before including sched.h
in my threading-related header file) I add -D_GNU_SOURCE to my compile command line,
now all of a sudden the compiler sees both definitions and says

	platform.h:804:0: warning: "_GNU_SOURCE" redefined [enabled by default]
	<command-line>:0:0: note: this is the location of the previous definition

...and the "implicit" warnings disppear. Anyway, add that one to the build-related mental "WTF?" bin
and just #define __USE_GNU instead.
*/
#define __USE_GNU
#include <sched.h>

#include <sys/sysctl.h>
#include <unistd.h>	// Needed for Posix sleep() command, among other things
#ifdef OS_TYPE_LINUX

	// These additional Linux-only includes make sure __NR_gettid, used in our syscall-based get-thread-ID, is defined:
	#include <linux/unistd.h>
	#include <asm/unistd.h>

#elif defined(OS_TYPE_MACOSX)

	// Mac OS X affinity API is via these:
	#include <mach/thread_policy.h>
	#include <mach/mach.h>
	// Gah - the osx mach/*h header tree sets its version of various CPU_IS_*** #defs,
	// incl. CPU_IS_SPARC, which then causes FP_MANTISSA_BITS_DOUBLE to get set to 53
	// rather the x86_64-required 64 ... that caused me to rename CPU_IS_* to CPU_IS_*.
#endif

#include <pthread.h>
// Found pthread header?
#if(defined(_PTHREAD_H))
	#define MULTITHREAD
	#define USE_PTHREAD
#else
	#error Pthreads header <pthread.h> not detected - Platform may not provide multithreading support.
#endif

And here is the threadpool.c snip the compiler barfs on under this particular Linux setup:

cpu_set_t cpu_set;
int i,errcode;
pid_t thread_id = syscall (__NR_gettid);

printf("executing worker thread id %u, syscall_id = %u\n", my_id, thread_id);

CPU_ZERO (&cpu_set);
i = my_id % pool->num_of_cores;	// get cpu mask using sequential thread ID modulo #available cores
CPU_SET(i, &cpu_set);
errcode = sched_setaffinity(thread_id, sizeof(cpu_set), &cpu_set);
printf("syscall_id = %u, setaffinity[%d] = %d, ISSET[%d] = %d\n", thread_id,i,errcode,i,CPU_ISSET(i, &cpu_set));
if (errcode) {
	perror("sched_setaffinity");
}

Any help would be appreciated.

[rogue]

Are the #includes in the wrong sequence?

[chalsall]

Originally Posted by ewmayer

(Note: The code in question compiles fine under OS X, debian and ubuntu - I've no convenient interactive access to a multicore setup running RH).

Any help would be appreciated.

I would be very happy to provide you and yours with unprivileged access to multicore CentOS environments (effectively the same as RedHat).

$ cat /proc/version
Linux version 2.6.32-358.2.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Mar 13 00:26:49 UTC 2013

You won't be able to do anything as root, but you would otherwise have full usage. Additionally, I'd be willing to assist to install any "root-only" software needed.

Please let me know if you are interested and if I can assist.

[xilman]

Originally Posted by chalsall

I would be very happy to provide you and yours with unprivileged access to multicore CentOS environments (effectively the same as RedHat).

$ cat /proc/version
Linux version 2.6.32-358.2.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Mar 13 00:26:49 UTC 2013

You won't be able to do anything as root, but you would otherwise have full usage. Additionally, I'd be willing to assist to install any "root-only" software needed.

Please let me know if you are interested and if I can assist.

I can offer the same under Fedora. Fedora is the proving ground for subsequent RH releases.

In fact, Ernst already has access here under Fedora.

[pcl@anubis ~]$ grep mayer /etc/passwd
mayer:x:1002:1002:Ernst W Mayer:/home/mayer:/bin/bash

[ewmayer]

Originally Posted by xilman

I can offer the same under Fedora. Fedora is the proving ground for subsequent RH releases.

In fact, Ernst already has access here under Fedora.

[pcl@anubis ~]$ grep mayer /etc/passwd
mayer:x:1002:1002:Ernst W Mayer:/home/mayer:/bin/bash

Yes, I realized that last night on re-perusal of my code - you can see Fedora-specific notes I made in the pltaform.h snippet I posted, related to similar include-issues I ran into when first trying out the pool code on your system, before I had my own multicore linux box to play with. I suspect the issue the above user reports under RH is similar, but it seems not identical, otherwise the Fedora-related hacks I made should work for him.

Originally Posted by chalsall

I would be very happy to provide you and yours with unprivileged access to multicore CentOS environments (effectively the same as RedHat).

Thanks - but I think we cam make it easier than that - I have hacked out a small test-infrastructure and attached the resulting *tgz. You just need to build one file:

gcc -DUSE_THREADS th*c

(I tested it under OS X just now ... you may need to append -lm under Linux, but first try as above.)

Now, under the same RH scenario I noted, the include-file part should fail, and no binary result. FWIW, Sample output for a working binary:

ernst-w-mayers-macbook:TEST ewmayer$ ./a.out
Init threadpool of 4 threads
adding pool task 0
Setting CPU = 0 affinity of worker thread id 0, mach_id = 3843
Setting CPU = 1 affinity of worker thread id 1, mach_id = 4099
Setting CPU = 2 affinity of worker thread id 2, mach_id = 4355
adding pool task 1
Setting CPU = 3 affinity of worker thread id 3, mach_id = 4611
Return from Thread 0 ... adding pool task 2
Return from Thread 1 ... adding pool task 3
Return from Thread 2 ... Return from Thread 3 ... Done with all pool tasks.

Thanks in advance for the help, guys.

[TheJudger]

Hi Ernst,

Originally Posted by ewmayer

Thanks - but I think we cam make it easier than that - I have hacked out a small test-infrastructure and attached the resulting *tgz. You just need to build one file:

gcc -DUSE_THREADS th*c

on CentOS 6.4 (kernel 2.6.32-358.el6.x86_64 // gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)) it works?!

gcc -DUSE_THREADS th*c
threadpool.c:293: warning: ‘force_align_arg_pointer’ attribute ignored
/tmp/cc9VtwcJ.o: In function `threadpool_free':
threadpool.c:(.text+0x1174): undefined reference to `pthread_join'
/tmp/cc9VtwcJ.o: In function `threadpool_init':
threadpool.c:(.text+0x162b): undefined reference to `pthread_create'
collect2: ld returned 1 exit status

gcc -DUSE_THREADS -lpthread th*c
threadpool.c:293: warning: ‘force_align_arg_pointer’ attribute ignored

./a.out
executing worker thread id 0, syscall_id = 14863
executing worker thread id 1, syscall_id = 14864
executing worker thread id 2, syscall_id = 14865
 Init threadpool of 4 threads
adding pool task 0
syscall_id = 14865, setaffinity[2] = 0, ISSET[2] = 1
syscall_id = 14863, setaffinity[0] = 0, ISSET[0] = 1
syscall_id = 14864, setaffinity[1] = 0, ISSET[1] = 1
executing worker thread id 3, syscall_id = 14866
adding pool task 1
Return from Thread 0 ... syscall_id = 14866, setaffinity[3] = 0, ISSET[3] = 1
adding pool task 2
Return from Thread 1 ... adding pool task 3
Return from Thread 2 ... Return from Thread 3 ... Done with all pool tasks.

Oliver

[ewmayer]

Originally Posted by TheJudger

Hi Ernst,

on CentOS 6.4 (kernel 2.6.32-358.el6.x86_64 // gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)) it works?!

Thanks, Oliver - as Paul noted, anything that works under Fedora should work under RedHat, that's why this user's build issues puzzle me. I suspect there may be something odd/unusual about his setup.

[He's at the phys/astro dept. of the U. of Sussex, in case any of our readers in the UK happen to be there as well, or know someone who is.]

Just to be sure - you can compile using just 'gcc -c -DUSE_THREADS th*c' and only need -lpthread for the link step, right? [I often forget that when building on my mac, because gcc-for-os-x does need the flag to be explicitly added.]

[ewmayer]

Ah, just found the problem - the user ignored my very clear instruction below which accompanied the 2 threadpool files needed for || builds of the code - I didn't realize this until Oliver's note re. addition of -lpthread to the link step caused me to double-check the compile/link options in the build log files the user sent me over the past week:

I have attached the 2 threadpool files needed for pthreaded builds - add -DUSE_PTHREAD to the compile line and ...-lpthread to the link step.

Thanks for acting as my muses here, guys. Forest, meet trees, etc.

----------

p.s.: I believe the apt initialism for the problem user X encountered here is PBKaC [problem between keyboard and chair.]