How to Scrape Website Titles & Descriptions from URLs in Airtable

In this simple tutorial, we'll scrape website titles and meta descriptions from a list of URLs in Airtable. We'll use the Data Fetcher Airtable extension as well as microlink, and there is no need for any code.

Microlink can be used to scrape data such as titles and meta descriptions from any URL, and the basic plan is free. 

Create Title and Description Table

In your Airtable base, create a table called 'Websites/ URLs'.

Create a field called 'URL' (with type 'URL'), then add the URLs you want to scrape titles and descriptions from. Create two other fields, 'Title' and 'Description' (with type 'Long text'.

Create a new Grid view called 'Needs Info' and add the following conditions:

• The 'URL' field is not empty.

• The 'Title' field is empty.

• The 'Description' field is empty.

Install Data Fetcher

Next, install Data Fetcher from the Airtable marketplace. Data Fetcher is free to start using and can be used to import many different types of data from different APIs or websites into Airtable.

Once Data Fetcher has been added to your base, create a new account or sign in to your existing one using the 'Have an account?' button on the bottom left.

Alternatively, sign-up for Data Fetcher with your Google login.

Scrape a Title & Description from a URL in Airtable

Data Fetcher requests are how data is imported into Airtable. You can create multiple requests within one installation of Data Fetcher in an Airtable base.

Click on 'Create your first request' from the Data Fetcher home screen.

On the create request screen, for Application, select 'Microlink' to use the Microlink Airtable integration.

For Endpoint choose 'Extract meta information from URLs'.

Give your request a name, such as 'Extract Meta Data', and click 'Save & Continue'.

Note: We can use the Microlink API without the need for Authorization, but if you go over Microlink's free plan limits, you can buy a Microlink paid plan here. Set the Authorization type to 'Header' and enter your API Key to increase your limit.

For URL, we want to insert a reference to the values in our 'URL' Airtable field. To do this click the + button.

On the modal that opens, select your output Table. Then select 'URL' for Field and 'Needs Info' for Run for every record in view. Click 'Confirm'.

The URL field should now contain a reference to the table.

Click 'Save & Run'.

The Data Fetcher request will run, and the Response field mapping modal will open. This is where you can specify which data to import from your chosen URLs and how they map to fields in Airtable. 

Use 'Filter all' to deselect all the pre-selected fields, then select 'Title' and 'Description' to import. 

Click 'Save & Run'.

You'll now see that the webpage Titles and Descriptions have been added for every record in 'Needs Info'.

Automatic Microlink Airtable Integration

Currently, you'd need to manually run the Data Fetcher request to import any updates to the website titles and descriptions. Although you can run the Microlink Airtable integration automatically using Data Fetcher's scheduling feature.

To upgrade your account, In Data Fetcher, scroll to Schedule and click 'Upgrade'.

Choose a plan from the different options depending on your needs and enter your payment details.

Back in Data Fetcher, click 'I've done this'.

Under Schedule click '+ Authorize'.

A window will open and prompt you to authorize the Airtable bases for Data Fetcher to access.

We recommend selecting 'All current and future bases in all current and future workspaces' to avoid any issues with any unauthorized bases in the future.

Click 'Grant access'.

Back in Data Fetcher, Schedule this request will now be toggled to on.

Select a schedule for the Microlink Airtable integration to run. You can choose intervals of 'Minutes', 'Hours', 'Days' or 'Months', then Click 'Save'. Any changes to the website titles and meta descriptions will now import to Airtable automatically on your chosen schedule.

What is a Website Title?

A website title, or title tag, is an HTML element that specifies the content of an individual webpage. 

You can usually add or change a title tag in a website's CMS, or by editing the 'title' tag in your website HTML:

<title>Data Fetcher for Airtable</title>

The title tag shows in search engine results and is an important ranking factor for SEO as well as being useful for users.

A title tag appears in search engine results.

What is a Website Description?

A website description (meta description) is an HTML tag that provides a summary of an individual webpage's content.

You can usually add or change a website description in a website's CMS, or by editing the (meta description) tag in your website HTML:

<meta name="description" content="Connect Airtable to any application with no-code using the Data Fetcher Airtable extension. Fetch data from any API into Airtable." />

A meta description appears under the page title in search engine results.