The use of the term “callback” makes it seem like the code that the task runs is in the same codebase/process as the one that scheduled the task, but these are distributed processes across many machines - so how does the “CreateTask” RPC caller specify what code to run in the task?
edit: seems like it is up to the callback owners/authors to deploy and run their own task workers, which presumably implements the individual task logic:
The design is very intentional in driving an ownership model where lambda owners own all aspects of their lambdas’ operations. To promote this, all lambda worker clusters are owned by the lambda owners. They have full control over operations on these clusters, including code deployments and capacity management. Each executor process is bound to one lambda
Our product has an asynchronous scheduler that runs various types of “Jobs”. (This is all for a PHP-FPM based server)
We have “RemoteJob”a that run entirely on a separate server. The name of the job, and a message (including information about how to run the job) are passed the remote queue which will schedule the job based on priority and availability. Ideal for sending emails, making requests to external services, processing web hooks, updating records in the search engine, bulk deletion, etc.
We have “LocalJob” that run in the same process but are deferred until after the response to the client is finished sending. These can be used for simple things like preparing a payload for a remote job, small bulk deletion operations, sending in app notification to a large pool of users, etc.
Additionally we have a “CallbackJob” that may schedule out a bit of work and callback an API endpoint in the app when there is load available, either to return the value of the job or to trigger additional processing. Our current use essentially been to come back the app when load is available to recalculate some expensive cached values.
edit: seems like it is up to the callback owners/authors to deploy and run their own task workers, which presumably implements the individual task logic:
The design is very intentional in driving an ownership model where lambda owners own all aspects of their lambdas’ operations. To promote this, all lambda worker clusters are owned by the lambda owners. They have full control over operations on these clusters, including code deployments and capacity management. Each executor process is bound to one lambda