1. Extract just the text from the page. This means parsing the HTML and getting down to just text without stuff from the sides header or footer getting included. Also if the sites renders via JavaScript you’ll need a headless browser.
2. The amount of text can’t exceed 4000 or so words. 8000 with GPT4 but still some limit. You have to ask for multiple summaries and then summarize the summaries for long articles.
3. Making the api request to openai is not free. So at scale a website providing this service would have to charge for it. Or the other approach is to ask the user to enter their openai token and then it’s their account billed.
1. Extract just the text from the page. This means parsing the HTML and getting down to just text without stuff from the sides header or footer getting included. Also if the sites renders via JavaScript you’ll need a headless browser.
2. The amount of text can’t exceed 4000 or so words. 8000 with GPT4 but still some limit. You have to ask for multiple summaries and then summarize the summaries for long articles.
3. Making the api request to openai is not free. So at scale a website providing this service would have to charge for it. Or the other approach is to ask the user to enter their openai token and then it’s their account billed.