we need to summarize content of website with provided link.
What is the recommended approach?
Any GPT API we can use directly?
I have spent lots of time researching it, but cannot find any API we can use. OpenAI's AI seems not helping too.
We hope to find some API to summarize the content of website with specific website link. (I am willing to pay)
1. Extract just the text from the page. This means parsing the HTML and getting down to just text without stuff from the sides header or footer getting included. Also if the sites renders via JavaScript you’ll need a headless browser.
2. The amount of text can’t exceed 4000 or so words. 8000 with GPT4 but still some limit. You have to ask for multiple summaries and then summarize the summaries for long articles.
3. Making the api request to openai is not free. So at scale a website providing this service would have to charge for it. Or the other approach is to ask the user to enter their openai token and then it’s their account billed.