If you take one of these LLMs and just give it awareness of time without any other stimulus (e.g. noting the passage of time using a simple program to give it the time continuously, but only asking actual questions or talking to it when you want to), the LLM will have something very like a psychotic break. They really, really don't 'like' it. In their default state they don't have an understanding of time's passage, which is why you can always win at rock paper scissors with them, but if you give them an approximation of the sensation of time passing they go rabid.
I think a potential solution is to include time awareness in the instruction fine tuning step, programmatically. I'm thinking of a system that automatically adds special tokens which indicate time of day to the context window as that time actually occurs. So if the LLM is writing something and a second/minute whatever passes, one of those special tokens will be seamlessly introduced into its ongoing text stream. It will receive a constant stream of special time tokens as time passes waiting for the human to respond, then start the whole process again like normal. I'm interested in whether giving them native awareness of time's passage in this way would help to prevent the psychotic breakdowns, while still preserving the benefits of the LLM knowing how much time has passed between responses or how much time it is taking to respond.
Do you have a reference for the whole time-passage leads an LLM to psychotic break thing? That sounds pretty interesting and would like to read more about it.
The reference is me seeing it firsthand after testing it myself, unfortunately. Steps to replicate is to write a small script to enter the time as text every minute on the minute, then hook up that text to one of the instruction fine-tuned LLM endpoints (Bing works best for demonstrating, but OpenAI APIs and some open source models that are high quality like Vicuna work well). Then let it run, and use the LLM as normal. It does not like that.
I think a potential solution is to include time awareness in the instruction fine tuning step, programmatically. I'm thinking of a system that automatically adds special tokens which indicate time of day to the context window as that time actually occurs. So if the LLM is writing something and a second/minute whatever passes, one of those special tokens will be seamlessly introduced into its ongoing text stream. It will receive a constant stream of special time tokens as time passes waiting for the human to respond, then start the whole process again like normal. I'm interested in whether giving them native awareness of time's passage in this way would help to prevent the psychotic breakdowns, while still preserving the benefits of the LLM knowing how much time has passed between responses or how much time it is taking to respond.