✨ Instagram integration is live! Tutorial

How to Scrape Website Titles & Descriptions from URLs in Airtable

Nov 18, 2022Rosie Threlfall

In this simple tutorial, we'll scrape website titles and meta descriptions from a list of URLs in Airtable. We'll use the Data Fetcher Airtable extension as well as microlink, and there is no need for any code.

Microlink can be used to scrape data such as titles and meta descriptions from any URL, and the basic plan is free. 

Create Title and Description Table

In your Airtable base, create a table called 'Websites/ URLs'.

Create a field called 'URL' (with type 'URL'), then add the URLs you want to scrape titles and descriptions from. Create two other fields, 'Title' and 'Description' (with type 'Long text'.

title-description-scrape1.jpg

Create a new Grid view called 'Needs Info' and add the following conditions:

  • The 'URL' field is not empty.

  • The 'Title' field is empty.

  • The 'Description' field is empty.

title-description-scrape2.jpg

Install Data Fetcher

Next, install Data Fetcher from the Airtable marketplace. Data Fetcher is free to start using and can be used to import many different types of data from different APIs or websites into Airtable.

Once Data Fetcher has been added to your base, create a new account or sign in to your existing one using the 'Have an account?' button on the bottom left.

Alternatively, sign-up for Data Fetcher with your Google login.

Data Fetcher Sign Up

Scrape a Title & Description from a URL in Airtable

Data Fetcher requests are how data is imported into Airtable. You can create multiple requests within one installation of Data Fetcher in an Airtable base.

Click on 'Create your first request' from the Data Fetcher home screen.

Create your first request in Data Fetcher

On the create request screen, for Application, select 'Microlink' to use the Microlink Airtable integration.

title-description-scrape5.jpg

For Endpoint choose 'Extract meta information from URLs'.

title-description-scrape4.jpg

Give your request a name, such as 'Extract Meta Data', and click 'Save & Continue'.

title-description-scrape3.jpg

Note: We can use the Microlink API without the need for Authorization, but if you go over Microlink's free plan limits, you can buy a Microlink paid plan here. Set the Authorization type to 'Header' and enter your API Key to increase your limit.

microlink-authorization.png

For URL, we want to insert a reference to the values in our 'URL' Airtable field. To do this click the + button.

title-description-scrape6.jpg

On the modal that opens, select your output Table. Then select 'URL' for Field and 'Needs Info' for Run for every record in view. Click 'Confirm'.

title-description-scrape7.jpg

The URL field should now contain a reference to the table.

title-description-scrape8.jpg

Click 'Save & Run'.

title-description-scrape9.jpg

The Data Fetcher request will run, and the Response field mapping modal will open. This is where you can specify which data to import from your chosen URLs and how they map to fields in Airtable. 

Use 'Filter all' to deselect all the pre-selected fields, then select 'Title' and 'Description' to import. 

Click 'Save & Run'.

title-description-scrape10.jpg

You'll now see that the webpage Titles and Descriptions have been added for every record in 'Needs Info'.

title-description-scrape11.jpg

Currently, you'd need to manually run the Data Fetcher request to import any updates to the website titles and descriptions. Although you can run the Microlink Airtable integration automatically using Data Fetcher's scheduling feature.

To upgrade your account, In Data Fetcher, scroll to Schedule and click 'Upgrade'.

schedule1.jpg

Choose a plan from the different options depending on your needs and enter your payment details.

data fetcher upgrade options

Back in Data Fetcher, click 'I've done this'.

schedule2.jpg

Click 'Add API key' and copy and paste your Airtable API key. You can find your Airtable API key here.

Group 11(1).png

Enter your Airtable API key and click 'Save'.

save-airtable-api-key.png

Toggle Schedule this request to on.

schedule-data-fetcher-request.png

Select a schedule for the Microlink Airtable integration to run. You can choose intervals of 'Minutes', 'Hours', 'Days' or 'Months', then Click 'Save'. Any changes to the website titles and meta descriptions will now import to Airtable automatically on your chosen schedule.

schedule-a-sequence-intervals.png

What is a Website Title?

A website title, or title tag, is an HTML element that specifies the content of an individual webpage. 

You can usually add or change a title tag in a website's CMS, or by editing the 'title' tag in your website HTML:

<title>Data Fetcher for Airtable</title>

The title tag shows in search engine results and is an important ranking factor for SEO as well as being useful for users.

A title tag appears in search engine results.

title-description-scrape12.jpg

What is a Website Description?

A website description (meta description) is an HTML tag that provides a summary of an individual webpage's content.

You can usually add or change a website description in a website's CMS, or by editing the (meta description) tag in your website HTML:

<meta name="description" content="Connect Airtable to any application with no-code using the Data Fetcher Airtable extension. Fetch data from any API into Airtable." />

A meta description appears under the page title in search engine results.

title-description-scrape13.jpg

Related Posts

How to Import Videos from YouTube URLs in Airtable

How to Import Videos from YouTube URLs in Airtable

Oct 6, 2022

Rosie Threlfall

microlink
How to Convert URLs to PDFs in Airtable

How to Convert URLs to PDFs in Airtable

Feb 4, 2022

Andy Cloke

PDFmicrolink
How to Get Screenshots from URLs in Airtable

How to Get Screenshots from URLs in Airtable

May 10, 2021

Andy Cloke

microlinkScreenshot