Making code 64 bit clean

Posted 4 Jun 2000 at 01:20 UTC by penguin42 Share This

As a Linux/Alpha user I often have the problem of getting code to work on my 64 bit machine. This article offers some examples, suggestions and warnings on how to make sure your code is 64 bit clean.

Making code 64 bit clean

(C) 2000 Dr. David Gilbert - linux@treblig.org. You may freely copy this document as a whole while keeping this copyright message. You may modify it as long as you keep the copyright notice and state that you have modified it. You may not charge for it.

4th June 2000. Please report any faults, comments or suggestions in this document

As a user of an Alpha processor running Linux I often face the problem of trying to get code to work on it which works fine on the x86 boxes most people are using. Often these programs use constructs which just don't work on 64 bit processors. While this might currently annoy a few nutty Alpha and SPARC users, the IA64 is about to land and mean that 64 bit systems become a bit more common. In this document I try and list some of the common things you have to be careful of to make sure that your code is 64 bit clean.

So what is different?

64 bit Linux machine use an organisation called LP64 where 'long's and pointers are 64 bit in length but everything else is 32 bit; long long's are also 64 bit. 32 bit Linux systems have both long's and pointers as 32 bit and have the 'long long' type as 64 bit.

Pointer lengths

The fact that a pointer is now longer than an int causes probably 80% of the problems encountered. The classic problems are people assigning pointer values to int's and then hoping the value can be converted back into a pointer at the other end. This just won't work on a 64 bit machine. If you need something which can reliably store either an integer or a pointer use a void*. Indeed this is what GNOME does - you'll see it casting int's back and forward to pointers to pass into event routines; when used properly this is safe (if a little unnerving - especially considering the number of compiler warnings that are generated!).

One interesting misuse of a pointer<->int cast I've seen is trying to increment a pointer by an integer offset. So say you are trying to pull apart a protocol and you have a pointer to a structure which you know is at pointer+n bytes where n is an integer. You could find:

struct mystructure* s=(struct mystructure*)(((int)p)+n);

Which works fine on a 32 bit machine. Unfortunatly the int cast is broken on 64 bit machines. A much cleaner solution is to use char*'s:

struct mystructure* s=(struct mystructure*)(((char*)p)+n);

One problem I have seen a few times is where an integer pointer offset is accidentaly made unsigned. So you end up with something like:

void sprang(char* ptr, unsigned int offset) {
  clunk(ptr+offset);
}

void anotherfunc(....) { sprang(p,-8); }

The compiler probably gives a warning here; but it probably stays unnoticed, and on a 32 bit machine due to the wonders of two-s complement arithmetic it does exactly what was expected.

On the 64 bit platform more 'interesting' things take place. The -8 gets converted to a very large 32 bit number and then is added to the pointer which ends up pointing into nomans land and kaboom!

Long abuse

So how long is a long? Well there is no really good definition of that in general! So if you start using a 'long' in place of an 'int' for some reason don't be surprised if it is a different size on a 64 bit platform. So if you are writing values to a hardware register then a 'long' probably isn't a good idea because it will be 64 bits on 64 bit Linux platforms and 32 bit on 32 bit Linux platforms. Other OSs may still keep long at 32 bits on 64 bit systems - so being portable can be difficult.

Misuse of longs can cause lots of problems. The simple ones are things like writing longs to files/sockets for set protocols/file formats. Suddenly you find an incompatibility between 32 and 64 bit machines. An 'int' is a nice reliable 32 bit size these days - so why go to longs? Answer - don't unless you are calling an OS or library routine which explicitly requests a long. If you are specifically looking for a 64 bit type then u_int64_t defined in sys/types.h is nice and explicit!

Interacting with varargs functions is fun with long's - consider this printf which works fine on a 32 bit machine but fails on a 64 bit machine:

long x;

printf("The value is %d\n",x);

Obviously a %ld was needed - but the lack of it didn't cause problems on 32 bits. So care must be taken with long's, in addition for pointers %p should always be used rather than %x (since this would have to be %lx on 64 bit systems).

Somethings are less obvious. For a start how do you print out the integer value of a system type (such as id_t) when you don't know if it is long or not? I can't see a truly safe way. Interestingly C++'s overloading solves all these problems rather nicely. One it does not solve is the problem of how many 0's to pad a pointer to when printing it!

Structure sizes

Another issue I'm not proposing a solution to is the sizing of structures. If you are trying to produce a structure of a nice binary size so that an array can be accesssed by shifitng rather than by multiplication you might put some dummy entries in to pad it. e.g.

struct athing {
  int x,y;
  char *next;
  int dummy;
}

On a 32 bit machine this structure would be a nice convenient 16 bytes in length; however on a 64 bit machine it turns out that it is 24 bytes and actually the best thing would be to leave dummy out.

Of select and ffs

The 'select' call returns its findings in an array of long's called fd_set. Many programs use the C library routine 'ffs' to find the first bit set in an element of the set. This is a fast way of finding out which of your file descriptors has come up trumps. Unfortunatly ffs only works on integers, not long's and hence you end up in the situation where if you have more than 32 file descriptors open you start ignoring some. Linux provides 'ffsl' in the glibc to help you with this, but it is not portable and the only sure way is to do it manually.


A few baisc tips for coders, posted 4 Jun 2000 at 02:57 UTC by aaronl » (Master)

1) Listen to compiler warnings. If you're using gcc, you should use -Wall, becuase while not always correct, compiler warnings can predict problems that might crop up. Most of the errors that were mentioned in this article produced compiler warnings.

2) Never use 'short', 'int', or 'long'. Since the code could be compiled on any platform, it is not obvious what size this data type will turn out to be. While it might seem like using a simple data type like long will not cause any problems, for example as a counter for a short 'for' loop that would never reach sizeof(long) cycles even on i386, using a type where you specify the size will be consistant across the platforms. When I talk about types with the size specified I mean system-defined typedefs like int16_t. Besides, why waste 4 more bytes if you know that you will not need 8 bytes for the variable, even on a 64-bit architecture?

3) Don't use C :). C is a great language which unfortunately does not have portability as one of its advantages. C/C++ are actually on the decline in the business world today, partially because of portability concerns. I admit, though, that it makes little sense to migrate to a different language just because pointer arithmatic and other unnecessary evils aren't completely portable.

Oops, found something else to say, posted 4 Jun 2000 at 03:00 UTC by aaronl » (Master)

> An 'int' is a nice reliable 32 bit size these days

Not really. When I programmed on the mac just a few years ago, the default size for int was 16 bits. Of course, longs were 32 bits. All the more reason to abandon types like 'int' and 'short' and switch to specific types such as int32_t.

C is portable, if you let it be, posted 4 Jun 2000 at 05:46 UTC by zw » (Master)

All my experience points just the opposite way from what you have said. You should avoid the types of known width (int32_t and the like) as much as possible. In fact, the only time to use them at all is when you have some external data format you have to interpret, or some algorithm defined in terms of arithmetic in N-bit fields.

What should you be using, then? Types with known properties. You can knock this down into just a few cases, for normal code:

  1. You need a plain old integer. Use int.
  2. You need to store the size of an object in memory. Use size_t.
  3. You need to represent the difference between two pointers. Use ptrdiff_t.
  4. You need a file offset. Use off_t.
  5. You need a time stamp. Use time_t - but never, ever write it into a file. Run it through gmtime() and write out broken down times in ASCII. Our children, dealing with the Y2038 bug, will thank you.
  6. You need a boolean. Use unsigned char in structures or globals, int in local variables. C99 has a real boolean type but it's not available widely yet.

In normal code, that's all you ever need. Note that not only do I not use the intN_t typedefs, I don't use short or long either. How does that fit with the original article, talking about problems with pointers being wider than ints? Simple. Don't ever stick pointers into ints. If you need to print them out for debugging, use the %p format, which expects a generic pointer. The dirty tricks people sometimes do with the low bits of pointers are best buried under unions and macros. Or not done at all.

The original article also mentioned the joys of structure packing. The best way to do that is to imagine you're working on a 64-bit platform when you lay out your structure, with the "natural" rules (LP64, all objects aligned to their size). The result will be laid out okay on 32-bit platforms, too. As a rule of thumb, put all the pointers first, then all the ints, then all the smaller stuff (if any). Don't worry about cache-line alignment, you'll just drive yourself mad that way.

Now, what about smaller stuff? C does give you short if you are concerned about chewing up memory. The key thing to remember here is that 32767 is a small number, likely to turn out to be too small. But you'll only find that out years later when it's too late to change things. Look at UIDs, most Unix implementations are slowly and painfully making them 32 bits wide. So think carefully before using short.

And then there's char, with its own horrid can of worms: whether or not it's a signed type is implementation-defined. Therefore, if you want an 8-bit integer, always explicitly specify signed or unsigned char. This doesn't come up very often; what does happen is that you have nasty bugs where a character is stored in an int, and gets sign extended on some platforms and zero extended on others - but you don't notice because all your test cases are 7-bit ASCII. I deal with this by using exclusively unsigned char for string data, and carrying around a header file full of wrappers for the functions in <string.h> that expect plain char.

It all sounds pretty awful - but it isn't, really. Just think of it as an incentive to write simple code. If you let it be, C can be portable. Certainly more portable than any of the alternatives.

zw makes no case againsts inttypes.h, posted 5 Jun 2000 at 16:04 UTC by Zaitcev » (Master)

I've read the reply by zw and while he advises against fixed size types he never tells us what is actually wrong. This is *all* he says about the matter:

    All my experience points just the opposite way from what you have said. You should avoid the types of known width (int32_t and the like) as much as possible. [...]

    Note that not only do I not use the intN_t typedefs, I don't use short or long either. [...]

So what wrong does happen when we use a fixed size type? Zw puts up a case against short, but that is largely irrelevant to fixed size types as such.

We want arguments. Until we hear them, we continue using <inttypes.h>, thank you very much.

why not use int17 or int32, posted 5 Jun 2000 at 19:25 UTC by Ankh » (Master)

If you use an int32 type, and someone ports your code to a 16-bit platform, what do they do? Did you use int32 because you need numbers larger than a specific value? What value? Or did you do it because you thought 32-bit integers were faster than 16-bit or 64-bit ones? Or because you're going to do bit manipulation and you have 31 flags?

If you use int, you're saying only that you want a whole number that can be positive or negative. If that's all you need, don't ask for something more precise, and your code is more portable as a result.

I have very rarely used a specific size in a type name; sometimes if you're on a 36-bit machine and a 32-bit int makes your program run five to ten times more slowly, you want to use 18 or 36 bit ints. And yes, I've used a 36-bit machine.

In general, you should make the simplest requests possible, not ask for something more specific than you need, and when yuo do ask for something specific, say why. This also means yuo should aviod reusing variables: if you have a single N to represent a person's age, someone might use an unsigned char fot it, but if they didn't notice you later use the same variable for their age in days, they're in trouble. Use int personAgeInYears; long personAgeInDays; and everything will be fine.

select and what not, posted 5 Jun 2000 at 23:30 UTC by jmg » (Master)

The use of ffs instead of the defined FD_{SET,CLR,ISSET,ZERO} is why code is not portable, not because people use the incorrect data types. If you follow the standards and not use any short cuts, then your code is much more likely to be portable.

This includes data types. Make sure that all of your data types match what the function asks for, and you'll see a alot more software work. Another bogosity that I see is that many people compare the return type of system calls (like read) against < 0 instead of == -1. When was the last time you checked and saw a man page read that it will return < 0 in case of an error? All the man pages will return -1 in case of an error. A syscall may come around where the < -1 values have meaning, and you'll end up assuming that it's an error.

It all comes down to knowning and following the standards that have been set. Not inventing your own because you "know" it's faster.

Why not to use int[XX]_t, posted 6 Jun 2000 at 13:59 UTC by eivind » (Master)

There are three reasons:

  • Speed. The basic C types are optimized for speed; forcing the use of other types can severely impact the speed of your program. On a true 64-bit architecture, using a 32-bit int could make your program mask out the upper 32 bits every time you do an operation.
  • Communication. By using forced types everywhere, you are communicating specific sizes even in the cases where they are not needed, thus making it hard for a programmer reading your program to know where they really ARE needed.
  • Portability. Forced types create an illusion of portability where it is likely to not really exist, and make it harder to find and fix the errors that really are there.

What is necessary to write portable code (with regard to sizes) is to know C. If you know C, you do not need forced sizes except in rare cases. If you don't know C, you won't be writing portable code anyway, and will think that your use of forced sizes help. Read the C FAQ, follow comp.lang.c and comp.std.c for a while, answering questions by referring to the C standard. In short order, you will have a feel for what is portable and what is not, and be able to write portable code.

Writing C that is portable to different type sizes isn't that hard - it just requires that you know the type restrictions and think a tiny bit about what you are doing. Using int[xx]_t everywhere does not help this.

Good coding isn't a black art, posted 7 Jun 2000 at 17:52 UTC by Rasputin » (Journeyer)

Most of what this article covers is just basic common sense. Taking one example, if the variable is defined as a long, then the printf should include a format specifier for a long. Pointers are not integers, to use another example, even though they are effectively interchangeable in many implementations if you assume a limited set of uses. These are two of many bad coding practices. In general, the more optimized your code is the less portable it will be (not to say that opimization is bad, but a lot of bad coding gets passed of as 'optimized'). Each level of optimization carries assumptions about when and how the code will be utilized. Many bad practices carry the same assumptions without the potential benefits. There are other trade-offs besides optimization, of course, such as maintainability, readability, etc that will impact this as well.

To comment on a comment elsewhere about the decline of the use of C, I have my doubts. The number of C programmers as a per centage of all programmers is declining, but I think that's more a function of people's expectations. A large number of companies want programmers, but there aren't that many good C programmers around, and it takes years to train one. On the other hand, a functional VB (to use the most prevalent example) programmer can be had at a fraction of the cost and training, which elevates their numbers while the number of C programmers remains relatively stable. When someone has successfully written and compiled a *nix variant in a language other than C, I'll reconsider my opinion ;)

In general, portable code, like good code, is not an accident. If you follow a few simple rules, and think about what you're doing before you do it, it's not that tough.

etc, posted 8 Jun 2000 at 01:50 UTC by mbp » (Master)

I think one reasonable place to use int32_t etc is in encoding network protocols defined to be particular lengths.

I'm in the habit of printing out size_t for example by something like this:

printf("allocated %ld bytes", (long) len);

The Samba group just had a donation of a new UltraSPARC. I think I'll try building rproxy there and see how pure I am.

Network stacks are unportable code, posted 8 Jun 2000 at 08:23 UTC by johnm » (Journeyer)

If you're using int32_t in a network stack to get an int of a particular length, it's probably because you want to block read a whole header, or cast a struct over a block you've read in.

struct tcphdr {
  u_int16_t th_sport;
  u_int16_t th_dport;
  ...

This kind of code is inherently unportable. You might have to play games with the endianness, and there's no reason to believe that the compiler won't insert padding into the struct so that the fields aren't where you think they are. But you can write the code with a particular architecture in mind -- and maybe even a particular compiler, so you can use pragmas to control the struct layout -- and it will probably be pretty fast.

It is possible to write this sort of code portably. To write the code so it's independent of endianness and struct layout, you can manipulate the individual bytes directly:

unsigned char *raw_buffer;
h.th_sport = (raw_buffer[0] << 8) | raw_buffer[1];
h.th_dport = (raw_buffer[2] << 8) | raw_buffer[3];

But at this point you don't need control over the size of the fields: they don't need to match the wire format exactly, they only need to be big enough. You might as well just use plain old unsigned int, which is guaranteed by the Standard to be big enough.

I get obsessed about struct layout and alignment control, because I've seen it abused a lot. I think types like int32_t help give the illusion that code like that described above is portable when really it is not.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page