On: Web Scraping 101 with Python

[philshem]

Before jumping into frameworks, if your data is lucky enough to be stored in an html table:

    import pandas as pd
    dfs = pd.read_html(url)

Where ‘dfs’ is an array of dataframes - one item for each html table on the page.

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

[kortex → philshem]

Sometimes it's also helpful to use beautiful soup to isolate the elements you want, feed the text of the elements into StringIO and give that to read_html.

[philshem → kortex]

Yes, this is a good idea for more complicated cases.

[mritchie712 → philshem]

We made a chrome extension that queries any html table in any open tab with SQL:

https://chrome.google.com/webstore/detail/sqanything/naejbcf...

You can export the results to Google Sheets too. One advantage of the extension is it works with JS rendered tables.

[Yenrabbit → philshem]

Handy! I'm also a big fan of pd.read_clipboard() for specific selections.

[antisthenes → philshem]

Holy crap, is there anything pandas can't do?

[Frost1x → antisthenes]

Ingest bamboo? Sorry, couldn't resist.

[JosephRedfern → philshem]

Woah. I’ve used pandas a fair amount and had no idea about this. Thank you!

[andreilys → philshem]

+1 this has saved me countless of hours

[mikesholiu → philshem]

what does this do

[SirSourdough → mikesholiu]

It reads HTML and returns the tables contained in the HTML as pandas dataframes. It’s a simple way to scrape tabular data from websites.