How to Scrape Website Titles & Descriptions from URLs in Airtable

Nov 18, 2022‚ÄĘRosie Threlfall

In this simple tutorial, we'll scrape website titles and meta descriptions from a list of URLs in Airtable. We'll use the Data Fetcher Airtable extension as well as microlink, and there is no need for any code.

Microlink can be used to scrape data such as titles and meta descriptions from any URL, and the basic plan is free. 

Create Title and Description Table

In your Airtable base, create a table called 'Websites/ URLs'.

Create a field called 'URL' (with type 'URL'), then add the URLs you want to scrape titles and descriptions from. Create two other fields, 'Title' and 'Description' (with type 'Long text'.

title-description-scrape1.jpg

Create a new Grid view called 'Needs Info' and add the following conditions:

  • The 'URL' field is not empty.

  • The 'Title' field is empty.

  • The 'Description' field is empty.

title-description-scrape2.jpg

Install Data Fetcher

Next, install Data Fetcher from the Airtable marketplace. Data Fetcher is free to start using and can be used to import many different types of data from different APIs or websites into Airtable.

Once Data Fetcher has been added to your base, create a new account or sign in to your existing one using the 'Have an account?' button on the bottom left.

Alternatively, sign-up for Data Fetcher with your Google login.

Data Fetcher Sign Up

Scrape a Title & Description from a URL in Airtable

Data Fetcher requests are how data is imported into Airtable. You can create multiple requests within one installation of Data Fetcher in an Airtable base.

Click on 'Create your first request' from the Data Fetcher home screen.

Create your first request in Data Fetcher

On the create request screen, for Application, select 'Microlink' to use the Microlink Airtable integration.

title-description-scrape5.jpg

For Endpoint choose 'Extract meta information from URLs'.

title-description-scrape4.jpg

Give your request a name, such as 'Extract Meta Data', and click 'Save & Continue'.

title-description-scrape3.jpg

Note: We can use the Microlink API without the need for Authorization, but if you go over Microlink's free plan limits, you can buy a Microlink paid plan here. Set the Authorization type to 'Header' and enter your API Key to increase your limit.

microlink-authorization.png

For URL, we want to insert a reference to the values in our 'URL' Airtable field. To do this click the + button.

title-description-scrape6.jpg

On the modal that opens, select your output Table. Then select 'URL' for Field and 'Needs Info' for Run for every record in view. Click 'Confirm'.

title-description-scrape7.jpg

The URL field should now contain a reference to the table.

title-description-scrape8.jpg

Click 'Save & Run'.

title-description-scrape9.jpg

The Data Fetcher request will run, and the Response field mapping modal will open. This is where you can specify which data to import from your chosen URLs and how they map to fields in Airtable. 

Use 'Filter all' to deselect all the pre-selected fields, then select 'Title' and 'Description' to import. 

Click 'Save & Run'.

title-description-scrape10.jpg

You'll now see that the webpage Titles and Descriptions have been added for every record in 'Needs Info'.

title-description-scrape11.jpg

Currently, you'd need to manually run the Data Fetcher request to import any updates to the website titles and descriptions. Although you can run the Microlink Airtable integration automatically using Data Fetcher's scheduling feature.

To upgrade your account, In Data Fetcher, scroll to Schedule and click 'Upgrade'.

schedule1.jpg

Choose a plan from the different options depending on your needs and enter your payment details.

data fetcher upgrade pricing plans.png

Back in Data Fetcher, click 'I've done this'.

schedule2.jpg

Under Schedule click '+ Authorize'.

Schedule Data Fetcher

A window will open and prompt you to authorize the Airtable bases for Data Fetcher to access.

We recommend selecting 'All current and future bases in all current and future workspaces' to avoid any issues with any unauthorized bases in the future.

Click 'Grant access'.

schedule-authorize-bases.jpg

Back in Data Fetcher, Schedule this request will now be toggled to on.

Select a schedule for the Microlink Airtable integration to run. You can choose intervals of 'Minutes', 'Hours', 'Days' or 'Months', then Click 'Save'. Any changes to the website titles and meta descriptions will now import to Airtable automatically on your chosen schedule.

Schedule this request

What is a Website Title?

A website title, or title tag, is an HTML element that specifies the content of an individual webpage. 

You can usually add or change a title tag in a website's CMS, or by editing the 'title' tag in your website HTML:

<title>Data Fetcher for Airtable</title>

The title tag shows in search engine results and is an important ranking factor for SEO as well as being useful for users.

A title tag appears in search engine results.

title-description-scrape12.jpg

What is a Website Description?

A website description (meta description) is an HTML tag that provides a summary of an individual webpage's content.

You can usually add or change a website description in a website's CMS, or by editing the (meta description) tag in your website HTML:

<meta name="description" content="Connect Airtable to any application with no-code using the Data Fetcher Airtable extension. Fetch data from any API into Airtable." />

A meta description appears under the page title in search engine results.

title-description-scrape13.jpg

Related Posts

How to Import Videos from YouTube URLs in Airtable

How to Import Videos from YouTube URLs in Airtable

Oct 6, 2022

‚ÄĘ

Rosie Threlfall

‚ÄĘmicrolink
How to Convert URLs to PDFs in Airtable

How to Convert URLs to PDFs in Airtable

Feb 4, 2022

‚ÄĘ

Andy Cloke

‚ÄĘPDFmicrolink
How to Get Screenshots from URLs in Airtable

How to Get Screenshots from URLs in Airtable

May 10, 2021

‚ÄĘ

Andy Cloke

‚ÄĘmicrolinkScreenshot