FFI is a mechanism by which a program written in one programming language can call routines or make use of services written in another. As far as we know, client can invoke RESTFul API via HTTP to call web server. So is RESTFul paradigm or HTTP a form of FFI?
Ivory to tower statement: Architecture calls these components and connectors. A component can be something as small as a function or as big as a monolith. A connector specifies the connection and its attributes between two components.
However, connectors can be of any form. They can be based on memory/processor (like local function calls), based on network protocols (like http calls), databases (a system a drops a record another one picking it up), REST (changes to collections) etc. Also some connectors imply higher level concepts such as response and request, message queues etc.
There is literally science about connectors.
IMHO, FFI is a abbreviation used in programming language design to specify the capability to invoke functions not specified by the ABI of the programming language (so the connector CPU/memory/in-process/same-os).
If your perspective/abstraction is that you are a SaaS doing some external integration, this generic rest/http API surely feel like FFI but I would not kidnap a term used for language/runtime design.
Why the derail about Architecture: the words you search are in the science behind Architecture.
Your comment just gave me a brain thing and it’s probably obvious to smart people but here goes:
Connectors are what really matter when it comes to de-risking a project. They’re where you delineate things that are nice to haves. They give you escape hatches (like your data popping out of one component in a format that can be edited by hand, stored, loaded). They let you more trivially replace and upgrade pieces.
Components aren’t all that interesting. They might be a place for optimization but mostly are just implementation detail.
And at any scale I think about in my work, there’s components and connectors. Whether an HTTP API or a function signature.
Is this what the Smalltalk guy meant by it being all about message passing?
> Connectors are what really matter when it comes to de-risking a project.
Yes, they define the architectural style and thus what architectures you can build.
This is getting more and more important because the default architectural style of the vast bulk of our programming languages, call/return, is unsuited for build most of the types of systems we are trying to build.
> Components aren’t all that interesting.
Yes and no. The architecture is defined by the connectors, but in day-to-day programming, your functionality is going to be mostly in the components.
> And at any scale I think about in my work, there’s components and connectors
Yes.
Which is why I created an architecture-oriented programming language that lets you deal with components and connectors
Connectors cover many non functional requirement attributes which like you point out de risk the project.
Components and their behaviors is where the functional requirements are and also many non functional requirements are handled. There is tons of architecture and design space when it comes to e.g. the SOLID patterns (applied to components not objects) and the onion architecture comes into play. So there is tons of interesting stuff to do with components.
About derisking: the core pillar of architecture when it comes to Microservices is finding the right bounded context. This massively de-risk projects as well. It is not so much technical architecture like the mentioned connectors but analysis of the requirements.
Connectors are the primary manifestation of software complexity. This doesn’t mean connectors are bad. They are necessary.
If they are uniform and well understood, then you get to reason about them more easily. That is part of the idea of „the smalltalk guy“ that you refer to. Hint: there is not just one guy.
You are correct. When you develop a FFI, you solve many of the same issues that are solved in RPC (remote procedure calls).
When you convert Lisp objects to C objects and vice versa across the call, this is very similar to marshalling and unmarshalling remote procedure call arguments.
The different representations and memory management of the objects are such that the two domains are often like different machines.
If you use FFI in reverse, you could use it to implement RPC. So that is to say, you can register a FFI callback in some C code which will pass some structure into the Lisp image. It gets converted to a Lisp structure. But what is that? Marshaling. The Lisp code could pass it over a network as an S-expression (further marshalling). The other side receives it, converts it to a Lisp structure, and then calls a FFI function into C code, where that becomes a C structure. Thus a RPC was achieved, leveraged by FFI.
The minimum requirements to legitimize the use of "foreign function interface" is that one domain activates a call in another domain, communicating data flows, which undergo a change in data representation.
The consideration of different programming languages is a red herring! Two different languages could have a very similar native representation. For instance, if we have Pascal and C modules directly calling each other because they use the same calling conventions and
data representations like "record of two 32 bit integers", that is not FFI. C calling assembly routines and vice versa isn't FFI.
"Foreign" doesn't so much mean "Foreign language" but "Foreign run-time/VM/data model.". Under the hood it might all be x86 machine code, which is the same language!
Typically FFI denotes the coupling of some managed environment with something closer to the metal, but other situations/combinations can fit the concept.
(Claimer: I've authored a well-developed FFI, as well as more than one remote procedure implementations, with data-description-driven marshaling/unmarshaling of binary objects. So you don't have to take anything here with any salt.)
There should be an interface description language that works equally well for FFI and RPC. It could be agnostic of the transport mechanism and the type system used to describe messages, and merely enumerate possible methods in a hierarchical (and composable) way.
Where we see a difference between the two is that FFIs are designed to be able to directly call all sorts of uncooperative code which is oblivious to FFI, without adding any wrapper code to it. FFI languages have a lot of detail because of that, for describing binary types, calling conventions, memory management ownership and such. (You can use the binary types system of the FFI even if you aren't calling any foreign functions; e.g. to parse a header in a binary file.)
Whereas in RPC, we usually control both sides of the call. If there is third party code, we will glue to that from the server-side RPC stub. Even if this is a RPC based on a lower level language like C, it doesn't have to have the generality of describing any C type or function; you design your RPC API to the limitations of the RPC system. E.g. if, say, it cannot pass structures by value, you don't do that in the API. If it doesn't handle bit fields in structures, you likewise don't feature such a thing in your API.
There is, it's called COM, and Windows as an operating system and ecosystem is built on top of it. (Though COM as a specification is not limited to Windows)
The language you use for defining the types is creatively named "Interface Definition Language (IDL)", and the "MIDL" compiler converts the interface definitions into C & C++ types and objects which can be used. Both via in-process/inter-process function calls, or via RPC/DCOM (Distributed COM).
"As a cross-language component model, COM relies on an interface definition language, or IDL, to describe the objects and associated functions. The COM IDL is based heavily on the feature-rich DCE/RPC IDL, with object-oriented extensions. Microsoft's own implementation of DCE/RPC, known as MSRPC, is heavily used as the primary inter-process communication mechanism for Windows NT services and internal components, making it an obvious choice of foundation."
"DCOM (Distributed COM) extended the reach of COM from merely supporting a single user with separate applications communicating on the Windows desktop, to activating objects running under different security contexts, and on different machines across the network. With this were added necessary features for configuring which users have authority to create, activate and call objects, for identifying the calling user, as well as specifying required encryption for security of calls."
If you want to learn more about this (it's an incredibly powerful concept that has a wide range of uses and is very much still used today), my recommendation would be:
1. The book: "Essential COM (1997)" by Don Box, one of the architects of COM at Microsoft in the 90s
2. The online tutorial "COM in plain C", which is one of the only (it might even be, like, THE only) resources for learning how COM actually works, since generally you use C++ frameworks to interact with it and they cover up all the details.
Disclaimer: I'm no expert in COM myself, I'm also working through these same learning materials.
I have a strong interest in FFI/Interop between languages and so have been working through things like "What is an ABI, conceptually?", understanding the C++ Itanium ABI, reading about vtable's, manually emulating C++ classes and class-inheritance with C structs of function pointers (vtables) etc.
COM is foundational here so it's the thing on my list atm. Hope you find this useful!
Conceptually, no, as others have said. The various RPC protocols- some of which can be inefficiently layered on top of HTTP- are FFIs.
FFIs are conceptually simple- call and response on the wire with a schema ABI.
The conceptual model for HTTP is much more complex. It is for synchronizing portions of a collection of data models between two systems.
HTTP includes expectations for an extensible set of data types that may be transferred, as wholes or in parts. These data types have identities and a variety of encodings and representations, and they may occasionally be taxonomically relocated, or may disappear altogether.
HTTP includes a robust caching layer and support for both transactional and streaming interactions.
In HTTP/2 it allows the client to request many data types at the same time and for the server to optimize for delivery even for elements the client has not yet requested but the server knows it will need.
The acronym REST- Representational State Transfer- captures the conceptual goal for the HTTP protocol. A client/server system has complex state on both sides, and needs a protocol that is aware of the complexities of transferring that state. It is much richer and deeper than FFI.
Not being an expert on the topic, to me FFI implies a tighter coupling between caller and called, typically that they will be in the same memory space.
I would consider RESTful and HTTP APIs closer to remote procedure calls (RPC).
RPC's can be abstracted so that you don't know. E.g Microsoft COM: same call to call an in-process COM object versus DCOM. Within the process, it could be "apartment threaded": the other domain has its own thread, which receives the argument material as a request, does it, and replies. (Kind of thing). Or it could be direct: your thread calls the function.
I'm sure people will tell you "No, that's not true FFI."
It's close enough, if you generalize REST to a sort of RPC where HTTP is the transport/protocol.
There are problems you run into when you use "true" FFI -- namely ABI/binary compatibility (you're beholden to compiler or OS behaviors) and the fact that non-native languages can't directly communicate with you.
Doing something over the network has a performance penalty, and there's marshalling costs involved.
But IMO, if you expose an IPC or socket-based API (rather than a C ABI) I'm not sure very many people could fault you.
A while back it occurred to me that one of the main benefits of microservices is that each team, and even each service, can have its own language/stack. Beyond accommodating people's preferences this allows the organization to experiment with different languages more easily. All they need is an HTTP library
So yeah, I'd agree that it's a kind of FFI. Or maybe even a "lingua franca"
However, connectors can be of any form. They can be based on memory/processor (like local function calls), based on network protocols (like http calls), databases (a system a drops a record another one picking it up), REST (changes to collections) etc. Also some connectors imply higher level concepts such as response and request, message queues etc.
There is literally science about connectors.
IMHO, FFI is a abbreviation used in programming language design to specify the capability to invoke functions not specified by the ABI of the programming language (so the connector CPU/memory/in-process/same-os). If your perspective/abstraction is that you are a SaaS doing some external integration, this generic rest/http API surely feel like FFI but I would not kidnap a term used for language/runtime design.
Why the derail about Architecture: the words you search are in the science behind Architecture.