Save a list into a table

In this guide we read every story on the Hacker News front page and save each one into a table: its title, its link and its id. Run it again and nothing is saved twice. This is how you collect products, posts, prices or leads from any listing page.

What you build: Manual → Open URL → Extract (list) → Loop over items → Data table (back into the loop), then Notify when done

The website: https://news.ycombinator.com. Each story is a table row <tr class="athing" id="…"> with its title link in .titleline > a.

Step 1: Make the table

The table keeps your data between runs.

1. In the sidebar, open Dataset › Tables.
2. Press New table.
3. Name: Stories.
4. Under Columns, add title (Text) and url (Text).
5. Save it.

Every row also has a key of its own. We'll use the story's id as the key, so the same story can't be saved twice.

Step 2: Create the flow and open the page

1. Studio › Flows › New flow, name it Save stories, then Create.
2. Add first step › Manual.
3. + on Manual › Open URL. In URL, type https://news.ycombinator.com.

Step 3: Read every story

1. + on Open URL's Next › Extract.
2. Mode: List of items.
3. Element: tr.athing. That's one story row.
4. Under List fields, add three fields:

5. Limit: 30 (the front page shows 30).

Right-click Extract › Run from here (or run the flow with Test run) and open its Output. You'll see one item holding a list:

{
  "items": [
    { "id": "41923456", "title": "Show HN: …", "url": "https://example.org/…" },
    { "id": "41923311", "title": "…", "url": "https://news.ycombinator.com/item?id=41923311" }
  ]
}

Step 4: Go through the list one story at a time

Extract gives one item with a list inside. Loop over items turns it into one story at a time.

1. + on Extract's Next › Loop over items.
2. Go through: A list.
3. List: type {{ and pick Extract's items. It reads {{ nodes.extract.output.items }}.
4. Maximum items: 30.

Loop has two output dots:

• Item: once per story. Connect the steps to repeat here, and connect the last of them back into the Loop.
• Done: once, after the last story. Connect what comes after here.

Step 5: Save each story

1. + on Loop's Item dot › Data table.
2. Table: Stories.
3. Operation: Change or add. It adds a row, or updates the one with the same key, so you never get duplicates.
4. Where (column: value, or id):

{ "key": "{{ nodes.loop.output.item.id }}" }

5. Data (column: value):

{
  "key": "{{ nodes.loop.output.item.id }}",
  "title": "{{ nodes.loop.output.item.title }}",
  "url": "{{ nodes.loop.output.item.url }}"
}

6. Now close the loop: drag from Data table's Next dot back onto the Loop over items card's input dot.

nodes.loop.output.item is the story the loop is on right now.

Step 6: Say when it's done

1. + on Loop's Done dot › Notify.
2. Title: Stories saved.
3. Message: {{ item.count }} stories checked.

Your canvas:

Manual → Open URL → Extract → Loop over items ─Item→ Data table ─┐
                                   ▲                              │
                                   └──────────────────────────────┘
                                 └─Done→ Notify

Step 7: Test it, then run it for real

1. Press Test run. The loop card shows ×30 and the status card says Loop over items: item 12 of 30 as it goes.
2. Open Dataset › Tables › Stories. The rows are there, marked as test rows, kept apart from real ones.
3. Press Clear test data in the table to remove them.
4. Back in the flow, press Run and confirm Run for real?. This time the rows are real.
5. Press Run again. The rows are updated, not doubled: same key, same row.

Make it yours

The same five nodes work on any listing page. Change the URL, the row selector and the fields:

Need more than one page of results? Add Scroll (Until enough elements) before Extract on infinite-scroll pages. On numbered pages, open each page address in its own loop.

Next: .

FAQ

Why do I need Loop over items? Doesn't a node run once per item?

It does, but Extract gives one item with a list inside it, not one item per story. Loop over items hands the stories on one by one, and Done tells you when every story is through.

Why does the last step connect back into the loop?

That's how the loop knows a story is finished and takes the next one. If you forget the line back, the loop stops after the first story.

Can I save only new stories?

Yes. Before Data table, add another Data table with Find rows and Where { "key": "{{ nodes.loop.output.item.id }}" }. Its Nothing found dot leads to the save, and its Next dot goes straight back into the loop.

Did the test run write to my table?

Yes, but only as test rows. A test run keeps its rows apart, so it never touches real rows. Clear test data removes them.