Hacker News new | past | comments | ask | show | jobs | submit login
Modernizing C arrays for greater memory safety: a case study in the Linux kernel (kernel.org)
284 points by diegocg on Jan 31, 2023 | hide | past | favorite | 118 comments



> int flex[] __attribute__((__element_count__(items)));

While what the article describes is clever, it is needlessly complex, and filled with various compiler switches and extensions.

In contrast, here's a stupid simple approach:

https://www.digitalmars.com/articles/C-biggest-mistake.html

where bounds-checkable arrays are declared as:

    int a[..];
`a` consists of two fields, a `length` and a `pointer`. Indexing it means the compiler can (optionally) insert a bounds check it.

    int s[..] = "string";
    s[10] = 'x'; // fatal runtime error
We can turn a pointer into a bounds checked array by "slicing" it:

    int *p = (int*) malloc(10);
    int a[..] = p[0 .. 10];
A bounds checked array can be turned into a pointer:

    int *p = &a[3];  // point to 3rd element of a[..]
That's all there is to it. No pages and pages of compiler switches and extensions.

Does it work? We've been doing that with D for over 20 years. Hell yeah, it works. It works fantastically well. It does not disturb any existing C code.


That's nice, but attributes let us easily retrofit existing C code such as the Linux kernel in a way that supports multiple compilers and compiler versions. Just extensions, not compiler switches. And they don't muck with the ABI which is a requirement for stable kernel driver interfaces.

Also what you're proposing...would be an extension!


Reading the article, it doesn't look easy at all.

With the [..] proposal, it is easy enough to convert it back and forth between pointers and [..] to conform to required interfaces. One could even make the [..] implicitly convertible to a pointer.


Most of the article is devoted to complications related to handling arrays that are inline at the tail of existing structs that usually are layout and size sensitive. Hence the related two restrictions that the count variable keep the same name and location in the struct, and there is no explicit array head pointer (just the implicit location at which the array starts at the end of the struct).

The reasoning most likely being that a bunch of annotations to structs, and perhaps some changes to calls to kmalloc() would be less destructive and much simpler than breaking the kernel ABI and altering the base size of any struct that used that idiom while also having to change every for loop or whatnot in the kernel that uses an explicit counter member name.


Take a look at how we use __has_attribute in the kernel.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

That pattern allows us to gracefully support new compiler features optionally and gracefully when people upgrade to toolchains that support them.

I suspect with `..` we could do something similar with __has_feature preprocessor guards.

Whether it's these extensions or `..`, we still would need to update struct definitions. That's at least the same amount of work. If we need to update member names due to `..`, then that's even more work!

That said, the two production compilers that can build the Linux kernel already support member attributes, and compiler inserted bounds checks against immediates. It's trivial to key off an attribute to change the inserted bounds check to a runtime load of a the corresponding member (and maybe a min between that and a fixed length, for non-flexible arrays that may not have been fully initialized for instance) and check against that. With `..` I would need to add new tokens, parsing (both for the declaration, slicing, and implicit conversions), and then the codegen.

There are times where we don't care about ABI within the kernel, so I do think it makes sense to have two different extensions here (implicit fat pointers and non-ABI modifying ways of denoting existing struct members are meant to be runtime bounds). Deploying the correct variant will take careful thought I suspect.

I haven't put enough thought into converting to/from fat pointers, so I'm glad you brought that up. It's something I'll have to think about more.

My initial gut reaction to implicit conversions is that C has enough wretched implicit conversions and promotions that are error prone, but perhaps if it is such an ergonomics win...though having a fat pointer decay to a regular pointer _implicitly_ feels like what I never want to happen.


> having a fat pointer decay to a regular pointer _implicitly_ feels like what I never want to happen

D doesn't do that implicit decay, but I was thinking of the kernel requirements of no API change. A simple way to convert a phat pointer to a pointer is:

    &a[0]
which is used in D to interface with C code. Note that the syntax still works if `a` is a pointer!

Since the pointer <=> phat pointer conversions are trivial operations, one can easily go back and forth between them depending on what part of the code one wants the overflow checks on vs the legacy interfaces.

With a couple of C macros one could turn on/off declaring arrays as pointers or phat pointers, and turn on/off the slicing code. In this way it will still compile with old compilers.


Is https://dlang.org/spec/arrays.html the best place to learn more?


yes


Starting with "That's nice" is extremely flippant and is an immediate turn off to any subsequent statement. A stronger opening to discredit the [..] proposal would be to take the closing quote "We've been doing that with D for over 20 years" and point out that storing the length of an array of a certain type is a specialization to arrays of dependent typing that has been around since Howard and de Bruijn extended lambda calculus to match predicate logic by creating types for dependent functions and pairs. Each axis of the lambda cube is another source of nondeterminism that has to be reasoned with and languages that explicitly exclude said functionality for simplicity can point to as justification for such restrictions.


I didn't say D invented it, so no need to discredit it. I said D has been using it for 20 years and proves it works famously with C style pointers and arrays.


And no claim was made about what D invented. Continuous use does not prevent door latch bushings from disintegrating; How long some thing has been used for is not proof that the replacement will work.


That's nice, but this is a Wendy's.


Fat pointers are a purely value level construct. STLC alone covers that. Non-determinism isn't really related to the lambda cube, even CoC is deterministic.


A pointer type is understood "to point" at a memory address with no restriction on being stack allocated. CoC is ambiguous as many different versions of the theory of constructions have been formulated that are not all equivalent and not necessarily bounded & deterministic.


This isn't the same thing though. A flex array includes the data inline in the struct so allocating a struct with a flex array at the end requires just one call to malloc and avoids a pointer indirection when indexing into the array.


Bounds checked arrays do not have an extra indirection.


The question here is about what happens when such an array is part of a heap-allocated struct where there's already a layer of indirection. As far as I can tell, the [..] proposal would effectively be equivalent to pointer & length fields as far as data layout goes, but there are no pointers stored for a flexible array member, as getting the address of an element is just adding an offset to the struct's address, hence it has less indirection.


I like how this is going, but I'm missing a few things here:

> We can turn a pointer into a bounds checked array by "slicing" it:

> int *p = (int*) malloc(10); > int a[..] = p[0 .. 10];

When `p` is a parameter in a function, the function cannot know that it can create a slice of up to 10 elements (I assume that the `p[0 .. 10]` creates an array indexed from 0 - 9).

What if the line was:

     int a[..] = p[0..12]
Do we still get undefined behaviour?

> A bounds checked array can be turned into a pointer:

> int *p = &a[3]; // point to 3rd element of a[..]

Assuming that a indexes from 0 to 9, what happens when we use p with an out of range index, for example:

     int *p = &a[8];
     blah = p[3];
My main concern is how to tell other functions that the array has a maximum size, and how to determine (inside a function) what the maximum length of its parameters is.


> When `p` is a parameter in a function, the function cannot know that it can create a slice of up to 10 elements (I assume that the `p[0 .. 10]` creates an array indexed from 0 - 9).

That's right, when a bounds checked array is converted to a pointer, the bounds does not go with it. Presumably, the function receiving the p has some way to determine the length (such as strlen, or via another parameter) from which the correct array can be reconstructed by doing a slice.

> What if the line was: int a[..] = p[0..12] Do we still get undefined behaviour?

Yes, if the 12 extends past the end of the data p points to.

> Assuming that a indexes from 0 to 9, what happens when we use p with an out of range index, for example: int *p = &a[8]; blah = p[3];

You get undefined behavior.

> My main concern is how to tell other functions that the array has a maximum size

The same way it's done now, by strlen, passing another argument with the length, or the function is able to get the length by other means. When a bounds checked array is converted to a pointer, the bounds are not part of the pointer.


Thank you. I'm always pleased when I get a reply from WalterBright[1].

Some follow up questions:

1. If you could redesign the above mechanism, would you do anything differently?

2. In my Own Toy Language[2], I've toyed with the idea making all native arrays fat objects as it seems the best way to ensure that the compiler, at any point, as the ability to bounds check if necessary. All that goes out the window when you want to do FFI to some C function. Any thoughts on mitigating this?

[1] Sorta like telling people when you speak to some celebrity :-)

[2] Everyone designs their own perfect language; I'm no different.


> If you could redesign the above mechanism, would you do anything differently?

Nope. It was a home run.

> All that goes out the window when you want to do FFI to some C function.

And there you go! What D does, though, is bounds check the conversion of an array to a pointer, and then (in code marked @safe) not allow arithmetic on the pointer.

Additionally, D introduced the `ref` parameter which eliminates nearly all use cases of needing to pass by pointer to a function.


You aren't going to get a lot of love from C programmers by casting malloc() result...


We have bounds-checkable arrays already since C99:

int (p)[n] = malloc(sizeof p); (*p)[i] = 1; // run-time bounds check

https://godbolt.org/z/vb8dqx1od

But yes, having a type that included the bound makes sense. But I do not think using array syntax for pointers as in your proposal makes any sense.

Dennis Ritchie got it right: https://www.bell-labs.com/usr/dmr/www/vararray.pdf

Ritchie DM. Variable-size arrays in C. The Journal of C Language Translation 1990;2:81-86.


> We have bounds-checkable arrays already since C99

    void foo(int n, int (*p)[n]) {
      (*p)[n] = 1;
    }
which has failed to catch on, because it still stores the pointer and the length as two separately handled objects.

> Dennis Ritchie got it right

"This paper proposes to extend C by allowing pointers to adjustable arrays and arranging that the pointers contain the array bounds necessary to do subscript calculations and compute sizes."

It appears to be phat pointers.


It is unclear why it has failed to catch on. One reason probably was that Microsoft decided at one point that it will not support C99 because everything should program in C++ which put a damper on the adoption of newer C features. The other reason is that compilers did not actually supported bounds checking here (I added it in GCC for UBSan in 2015) and it is still possible to lose the bound at function calls (I have patch to GCC I hope I can submit for the next version). So for a long time most people thought that VM-types were useful only for numerics.

Having said this, I fully agree with you that having a wide pointer that combines length and pointer would be a useful feature. I just think that the syntax proposed by Dennis Ritchie fits naturally into C.


I had always assumed it was because of backwards compat that the proposal was never accepted but since it turns out that is not the issue, do you have any idea why the proposal was never accepted ?


Objective-C and C++ never had any issues having additionaly types for arrays and strings, C could do the same, but WG14 has clearly decided they don't want to do that.


Yes, so why WG14 or whoever in charge has not accepted that one simple addition responsible for so many errors ? It's not like it would affect performance. For strings, it would even make it better, simply moving the lenght around with the compiler's help.


Most likely as they have proven during the last 40 years, security is not in their agenda.

Even the Annex K design was misguided, as it expected parameters to still be separate pointer + length arguments, thus hardly changing anything regarding getting them wrong.


It turns out that bounds checked arrays are ideal for strings. One no longer has to run strlen, which is inherently slow and cache-unfriendly. One can also slice a substring without needing to allocate and copy.


The problem is that this would have one specific ABI, which probably wouldn't match many existing structs with a flexible array member at the end. Could potentially be used for new code (while requiring every user to upgrade the standard they compile with), but has the risk of not usable for modernizing old code.


You're right that bounds checked arrays do nothing at all for existing code. But they can be added incrementally to an existing code base, as a normal part of working on the code.

In that aspect it's like when prototypes were added to C. Nothing changed for existing code, but prototypes are so advantageous people would retrofit existing code incrementally when doing routine maintenance.


I presume prototypes could be added to transform much of existing code without compatibility problems (ABI nor API), which is a clear advantage compared to checked arrays. Checked arrays might worth it regardless, but it's not applicable in most of the article's use cases.


> Checked arrays might worth it regardless

I can attest that they are.

> it's not applicable in most of the article's use cases

I don't believe that.


For code that is critical to performance, C99's "flexible array at the end of a struct" is an useful tool. It basically allows you to attach a header at the beginning of some dynamically-allocated binary data of infinite length (yes, it can be implemented as a pointer at the end of the struct, but the extra latency of another pointer chasing can reduce performance). Before C99, the "size-1 hack" or "size-0 GCC extension" for this purpose was already widespread in both the Linux kernel and Windows [1], but with the disadvantage of triggering memory-safety tools, as the author pointed out.

Meanwhile, unlike C99, this construction is not allowed by any version of the C++ standards, any such use would be a non-standard extension, I think this is unfortunate. I only write C, I wonder if any C++ guru out there can answer this question: does modern C++ have a better solution to implement the same thing?

[1] https://devblogs.microsoft.com/oldnewthing/20040826-00/?p=38...


> Does modern C++ have a better solution to implement the same thing?

I'm no guru, but I know from experience you can do it in C++:

  {0}[calvin ~] cat test.cpp
  #include <iostream>
  #include <memory>
  
  struct foo {
    int len;
    int v[];
  };
  
  int main(void) {
    auto p = std::unique_ptr<foo>(reinterpret_cast<struct foo *>(
        malloc(sizeof(struct foo) + sizeof(int) * 2)));
  
    p->v[1] = 99;
    std::cerr << p->v[1] << std::endl;
  
    return 0;
  }
  {0}[calvin ~] g++ -Wall -Wextra -std=c++17 test.cpp -o test
  {0}[calvin ~] ./test 
  99
  {0}[calvin ~] clang++ -Wall -Wextra -std=c++17 test.cpp -o test
  {0}[calvin ~] ./test 
  99
EDIT: Remove unnecessary extern block, as pointed out by wahern in the replies.


Flexible array members are commonly supported as an extension in C++ compilers, but the C++ standard itself does not permit them. And FWIW `extern "C"` doesn't drop a C++ compiler into a "C mode", it merely effects linkage (i.e. name mangling); all the code within the scope must still be valid, compiler-supported C++. Your example code compiles the same without the extern "C" declaration. And it compiles the same if main is also placed within the extern "C" scope; but not the headers, as C++ constructs like templates cannot have C linkage.


Unrelated to the article but since you seem to know this stuff, do you know what happens when a function in an “extern C” block exposes C++ types (parameters or return types)?

I assume it’s broken because it’d expect the actual C++ types to be passed in despite the lack of ABI stability, and only the linking / name mangling is affected (possibly calling conventions as well?) but I’m not actually sure.


C++ types are structs and unions if you get down to the bottom of things. This is possibly not guaranteed by the standard, but AFAIK, extern "C" doesn’t ever affect the calling convention or memory layout for the parameters, only the name mangling for the linker. So in theory, if you manage to construct a valid std::string in C code (which is hard, given its invariants, and would be absolutely nonportable), then you could pass it to an extern "C" function that’s written in C++.


You could that with a function that would map a ptr/len pair into a std::span.


D'oh, thanks for pointing that out.


Yes, you can do this in C++, but keep in mind that implicitly generated copy/move constructors don't understand this and will not copy the full object. This can produce surprising memory corruption that can be difficult to debug. So you should be sure to either explicitly mark the struct not copy/movable, or implement smarter copy/move operators.


Tbf I don't think C copies the whole flexible array member either[0]. So the care you would normally take while using a flexible array member in C should also be taken in C++ :)

[0]: https://stackoverflow.com/questions/35423293/flexible-array-...


Right. But C has fewer scenarios where implicit copies are possible, in part because it does not have references.


It compiles and work doesn't mean it is not undefined behaviour. You should at least try with the ub sanitizer.

But in this case, as other said, it may be accepted as an extension. Still not part of C++


You can do it without trailing arrays, by stacking the structures after one and other:

MyStructA a; MyStructB b;

a = malloc((sizeof a) + (sizeof b)); b = (MyStructB *)&a[1];

You need to make sure that the second struct doesn't have stricter alignment requirements than the one preceding it, but using this technique you can stack any number of structures or arrays of structures in one allocation.

(I would generally not recommend this coding style unless you have very specific requirements of memory usage)


This is pretty similar to:

  MyStructA {
    ...
    MyStructB b[];
  };

  MyStructA* a = malloc(sizeof(MyStructA) + sizeof(MyStructB));
  b = &a->b[0];
(Except, of course, that the syntax for locating 'b' is nicer this way, because you don't have to explicitly address the memory after 'a' and cast it to 'MyStructB'.)


> does modern C++ have a better solution to implement the same thing?

No, the only standard way is to allocate a buffer large enough, and placement new the header and the bulk payload separately. The manual handling of alignment makes this very cumbersome.

Flexible array members would be nice, but I don't like that it is yet an other overload on some array declaration syntax (other meanings of "T x[]": 1) declare an array of unknown size, 2) in a definition, deduce the size). For some reason C likes to overload the array declaration syntax with widely different meanings (looking at you, VLAs).


> yes, it can be implemented as a pointer at the end of the struct, but the extra latency of another pointer chasing can reduce performance

For the latency to be significant digits, there must be a lot of repeated calls which only read/write a very small number of elements in the array. Otherwise the accumulation of read/write operations performed when iterating over the data would dwarf the single pointer dereference.

So I'm curious now-- do OS kernels spend most of their time doing lots of calls that dive into such dynamically-allocated data just to extract a single datum or two?


I think they were slightly off the mark. The bigger deal is avoiding an extra allocation for an entity that has a variable sized part.


While avoiding the allocation (and potential memory fragmentation) is probably nice, I bet the main benefit is memory locality.


Because C++ has offered better alternatives for bounds checked data structures since it exists.


No, C++ has nothing better.


here's a worse way that's a lot more work and maybe has some advantages:

https://gist.github.com/cozzyd/efda739301bb7eb3a4a63a145c93e...


Good article. If you're compiling C with MSVC then you can use SAL annotations [1] which serve the same purpose.

[1] https://learn.microsoft.com/en-us/cpp/code-quality/annotatin...


Wow, such a great annotation language. Wish it were in GCC as well.


I keep hoping that the C and C++ committees will get together and standardize some of that in the form of C23/C++11 style attributes. But that is sadly likely a naïve hope.


Microsoft hopes to be able to map SAL into C++ contracts if they ever be part of the standard, as that was their initial goal when they started implementing lifetimes support in VC++.

As for C folks, I don't have any hopes of them every going down that route.


Call me crazy, but zero length arrays are a great abstraction, when you working with implicit data-structures. Not safe, but elegant and performant. Many codebases could be 2x faster if their designers embraced that concept.


> Not safe, but elegant and performant.

I'd say it's also not more dangerous (or equally dangerous, depending on your camp) than a pointer from malloc().


both c and c++ have the concept of zero-length arrays, if you malloc them - int * a = malloc(0); is ok


I think the parent is talking about the c pattern of having the last member of a struct be a zero length array, which is actually a dynamically sized array that the struct is only the header to (ostensibly with another field of the struct specifying the length of the array). It's fallen a bit out of favor, but it is a handy way to commingle the header and array with one allocation/pointer.

And interestingly COBOL handled this in a cleaner way. I forget some of the specfics but there was a way to specify to the compiler that one field of a record specified the length of the following array, allowing the same pattern in a type safe way.


> And interestingly COBOL handled this in a cleaner way

That is the least reassuring sentence I have read on HN in a while!


ROFL I can use that.

- Boss!

- ..yes?!

- I was reading a COBOL code base and got a brilliant idea--

- Say no more, you need a vacation. I am sorry, I have been too pushy. Take 3 weeks; just promise me one thing: no COBOL.


In cobol it wasn’t dynamic, or infinite. The syntax of the array/table declaration includes a maximum and it would allocate the entire array statically up to the max.


According to the article, it's better to use the new flexible array syntax (int arr[]) instead of the old zero-length syntax (int arr[0]), because that allows the compiler emit better warning messages.


how would compiler confusion manifest between [] and [0] if it wanted to emit warning messages?


Technically, [0] is not allowed by the standard. The real issue is [] vs [1].


According to the C spec, zero length arrays are explicitly illegal.

> Zero-length array declarations are not allowed, even though some compilers offer them as extensions (typically as a pre-C99 implementation of flexible array members).

However, as they say, gcc (and therefore clang) have an extension that allows it. So does MSVC but it works slightly differently.


you should be specific about which C spec you are referring to, you talk about "the C spec" and then you mention the "pre-C99 implementation". Maybe you mean they've always been illegal in every version, but it would be more clear.


In reality, who cares of what the C spec says? Unless you have to port code on different compilers (which is very unlikely, unless you are building a library meant to be shared with different projects) you only care about the fact that the code works correctly with the compiler you choose to use.

I don't get all the programmers that scandalize if you use GNU extensions, they are fine, and mostly useful, so if you are using GCC to program I don't see why not use -std=gnu11 instead of -std=c11... who cares if the program is not compliant?

Even if you want to be compliant to share code, the C standard is you last problem, the thing is that you probably use a ton of libraries and header files specific to that implementation (such as POSIX or even worse Linux-specific stuff), so your code is 99% not portable anyway without rewriting most of it.


It’s all well and good until someone else wants to compile it on clang[1]. There’s a lot of value in C being a lingua franca of sorts, and that depends on a cross-implementation understanding of semantics, i.e. standards, (though that doesn’t have to be from whatever official body that produced C99 etc).

[1] IIRC on OSX[2] gcc actually invokes clang

[2] stopped upgrading before it became macOS


Clang implements nearly all GCC extensions. (No nested functions.)


The bit after `>` is a quote, not my words. See https://en.cppreference.com/w/c/language/array for the source

It's saying that C99 implemented flexible array members but before then some compilers introduced their own (nonstandard) implementation of flexible array members, using the (not allowed) zero sized array notation.


The sequence of event was basically:

1. Pre-C99, no flexible array was allowed.

2. Gradually, people started using the "size-1 array at the end of a sturct but write beyond" hack as flexible array.

3. As an attempt to do the hack in a more ordered manner, some compilers, including GCC, started officially supporting the non-standard "size-0" array extension.

4. C99 added flexible array with indefinite length (array[]), while prohibiting both the undefined-behavior "array[1]" and the non-standard extentios "array[0]".

So when people say "size-0" array, it could mean either of these three things, and it does get a bit confusing. But fundamentally the idea was the same, all of the three techniques are used to achieve the same practical effect.


Kind of:

> If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned to indicate an error, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object

So it may actually allocate (although the allocation is unusable).


Hang on, let me think this through...

If malloc(0) gets called as first malloc in the program the system break does not need to be moved, as there is always 0 bytes space available... but malloc does like to move sysbreak by a large amount at a time to reduce the need for repeated calls...

I'm guessing malloc(0) does not move sysbreak and simply returns a pointer to the bottom of the heap?


Implementation defined. I've heard of returning null (under the case that your free() implementation allows nulls to be passed in) or returning a pointer to a zero length object on the heap like you're suggesting. Really just about the only requirement is that the pointer can subsequently be given to free() since dereferencing the pointer is UB.


free(NULL) is required to be a no-op by the ISO C standard.


Oh, good call. I had that backwards in my memory. The issue is when you give out non NULL pointers to zero sized objects, you have to make sure to give out unique pointer bit patterns at least versus nonzero sized objects so that the matching calls to free don't stomp on eachother.


Just want to point out malloc(0) might gives you any pointer, and you still have to call free() on it.


malloc is a user level library function - c/c++ implementers can do what they like with it

assume the downvote was from someone that malloc is a system call


I downvoted for the assumption that sbrk is involved in any way. It's an implementation detail of (some) historical malloc implementations, not some inherent aspect of all implementations. The entire thought experiment is faulty.


Right, malloc uses mmap instead of sbrk, I'm an idiot. I really ought to read up on up-to-date implementation practices.


Then you have two allocations instead of one and related data is now likely to be farther away, possibly even in a different page.


malloc(0) return value is undefined by POSIX and can return NULL (IIRC it did on NetBSD).


malloc() doesn't allocate arrays. It allocates blocks of memory. Hence sizeof doesn't work the same for malloc() objects as it does on arrays.


sizeof doesn't work the same for malloc() because the type of the returned value is a pointer, and the behavior of sizeof is dependent solely on the static type. For comparison, calloc() is specifically defined as "allocates space for an array of ... objects" in the Standard, but since return type is still void*, the caveat with sizeof still applies.


That is not true, as long as VLA:s are in the language spec for the version you're using.

This works:

    void vla_print(int n)
    {
      int foo[n];
      printf("Got %zu bytes right there!\n", sizeof foo);
    }

    int main(void)
    {
       vla_print(47);
       return 0;
    }
This prints 188 [1].

Even if you "hide" n from the compiler, i.e. make its value something that is only known at run-time (which is jumping through hoops, pretty sure the above is enough).

Also, it was jarring that the fine article kept referring to sizeof as sizeof(), a notation that C programmers typically use to identify function names. As everyone knows, sizeof is not a function. It's an operator. I really need to print up that t-shirt soon. Or maybe my first tattoo ... Hm.

[1]: https://ideone.com/kKf3bT


In your example, sizeof works because the type of foo is int[n], so I'm not sure what point you're making. It's still true that the way sizeof works depends solely on the static type of the expression that it is applied to - if it's an array, you get the actual size, including the necessary dynamic computation if it's a VLA, and if it's a pointer, you get the size of a pointer even if it points to an array.


Well, sure, but the reason for that is that C's type system simply cannot represent functions returning arrays with known size. This is a weakness of the type system. Arrays degrade to a pointer type, lacking array length, as soon as you pass them between functions.


> sizeof is dependent solely on the static type

so what, an array has a size that sizeof can measure. what is returned from malloc is not an array and what sizeof measures is not reflective of the size of the allocation


calloc() doesn't allocate an array either. It's purpose is to allocate blocks of memory larger than SIZE_MAX. Mostly irrelevant now but was an issue on 16-bit systems.


I literally quoted the Standard (C17 7.22.3.2) in my earlier comment, and it very specifically says that calloc allocates an array. This language isn't from a new standard, either - it goes all the way back to ISO C90.


Does anyone know what is the status of their refactoring effort to update all the flexible array declarations in the kernel? How far along are they?


IMO the right approach is to start with counted array struct wrapper types like `struct array_of_xyz { unsigned count; xyz a[1]; };` and use them to hold and pass by reference. When the array sizes are fixed, then use `struct array5_of_xyz { xyz a[5]; };` and pass by reference or by value as needed. Add to this a decoration to indicate that the `count` field is a count of the number of elements in the array and now the compiler can do bounds checking.

Then fix codebases recursively until it's all ok. At ABI boundaries that don't use such types create values of such types corresponding to the given arguments (e.g., you could count the elements of `argv[]` then create a wrapper for the `argv`).


> Is it actually a 4 element array, or is it sized by the bytes member?

i give up, what does sizeof say? and why would it be sized by bytes?


The previous paragraph says

> ...due to yet more historical situations (e.g. struct sockaddr, which has a fixed-size trailing array that is not supposed to actually be treated as fixed-size), GCC and Clang actually treat all trailing arrays as flexible arrays.

But I don't know, that doesn't seem to match the result I am getting with clang 13.1.6. It does seem to respect the array size declared in the struct, not treat it as a flexible array. I get -Warray-bounds warnings if I try to access anything past o->variable[3]. Maybe I'm misunderstanding what they're saying or my example is screwed up.

Edit: Actually, I guess it does end up treating it like a flexible array -- it produces -Warray-bounds warnings when compiling, but the resulting binary works (and doesn't trigger asan). Not sure I entirely understand it though.


It treats them as flexible arrays in the sense that it doesn't assume indexing beyond the declared size is undefined behavior, which would have implications for code elision and other optimizations.


Thanks for the explanation! That makes sense.


Everyone says a memory-safe C would be slower, but has anyone actually tested that recently?

It seems that a memory safe C would be faster, in that you wouldn't have to learn yet another language and runtime to deploy your stuff.


> A simpler approach is the addition of struct member attributes, and is under discussion and early development by both the GCC and Clang developer communities.

Does anyone know where I can follow these discussions?


> C is not just a fancy assembler any more

I wish this trope would die. It really never was one.


Optimising C compilers maybe not, but you absolutely can naïvely translate C into assembler - at least for stack based machines. I do think ‘fancy assembler’ is a fitting description in that case.


In what sense was C never a fancy assembler?

I am not an expert on C nor assembly and would be curious if you could expand on this. The statement makes sense to me because my impression is that most of what happens in C code gets translated fairly straightforwardly to machine code, with the compiler taking care of bridging differences in the instruction sets of targeted architectures. I guess the reason this is simplistic is the inlining and loop unrolling done by an optimizing compiler. Is this what you mean?


The basic problem with this assumption is that people who follow it tend to get it in their heads that since C is merely "translat[ing] fairly straightforwardly to machine code", they assume they can rely on the semantics of the machine code being the semantics of their C program. That isn't true, and hasn't been for a long time [1]: compilers are only required to uphold the looser semantics of C, and they will happily apply optimizations that deviate from the semantics of a purported naive translation to machine code. The usual example brought out to explore this difference is signed integer overflow, which has nothing to do with inlining or loop unrolling.

[1] I don't know enough about the early history of C to be able to assert that it was never true, but it certainly hasn't been true since at least 1989.


I think compiler writers understand this "C is a portable assembler" entirely differently than regular compiler users, because compiler writes are in the trenches all day long and focus (too much?) on relatively minor code generation and memory model details which are just not relevant for most compiler users in their day to day work.

Of all the popular high level languages, C is still (among) the closest to CPU and memory, especially when taking compiler specific language extensions into account which are often simply not available in even higher level languages.

And while modern compilers do all sorts of funky and sometimes surprising code transformations in their optimizer passes, I can still look at a piece of C code and the compiler's output as assembly, and figure out which parts of the C code result in what parts of the assembly code. Some parts may be massively reduced, some parts expanded (e.g. by loop unrolling), some parts may have disappeared completely or shuffled around, but the relationship is still recognizable.

Also, from the POV of a programmer in the 70's or 80's who's used to manually stamping out separate programs for each CPU type in assembly, and then switches to C and only needs to write code once which then runs on different CPUs as if by magic, C would absolutely count as a "portable assembler", and I guess that's the origin of that phrase.


C is a language with a specification which defines it in terms of a virtual machine, not translation to machine code. The memory model is also totally different and it has lots of undefined behavior.


Care to explain?

Not saying C is a great standard, but that idea means that using C instead of assembly doesn't generate a big overhead, and it's still much easier to write C than assembly, especially when compiler support is very common.

I'm currently making a language that translates directly to C.


I have a couple of books of the days when C was relatively new that claim otherwise (around mid-1980's).


TL;DR is the introduction of C99 VLAs, not Pascal-style arrays, though a potential attribute could be added so we could do

  int some_int;
  int some_array[] __attribute__((__element_count__(some_int)));
to store the size of some_array in some_int.


This has nothing to do with VLAs. The article covers FAMs at the end of structs.


I must have misunderstood, thank you for the clarification. I thought that having no size specifier in the array declaration turned it into a VLA, which could have repercussions when embedding structs, e.g.:

  struct
  {
    int a;
    /* ... */
    int b[];
  } foo;

  struct
  {
    struct foo;
    /* ... */
    int c;
  } bar;
I would expect to have issues when trying to access the other members of struct bar like c.


You could take this idea further by adding an "array_ref" type to the C standard where it is a length and elements. In practice this would be broken into separate parameters when passed to a function so existing functions could be fixed post-hoc using attributes - similar to printf formatting attributes. Then passing an array_ref to a function that expects a pointer/length would let the compiler automatically translate. Then you could define the rules for how an array_ref decays to a pointer and how to make one out of a pointer and length.

In other words the C standard could make bounds-checking of arrays possible with a good interop story if the standards committee believed it was worth doing. Compilers would have a flag to enable or disable the runtime checks based on safety/perf tradeoffs. Libraries would slowly add the relevant annotations. Eventually most code would have the option of having all array accesses bounds checked.


Sibling already pointed out this article is not talking about VLAs, but I do like your flexible-array attribute extension proposal.


The proposal is almost exactly the same as the one in the article (near the end).


Yes, it is IMO the second most important part of the article, it seems like a good idea and it would be nice to see it in the standard.


Since clang will never support VLAIS, your proposition is a non-starter.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: