Medium articles all have "Just a moment...medium.comVerify you are human" for content

[jstoops]

Does anyone know how to crawl articles from Medium without getting this message? Might need to change the pipelines to get the articles in a different way as I assume Medium put this in place to specifically prevent bots crawling the public articles in this way.

I'm going to try to change it to use this API and will post back how it goes. If anyone has a better solution please let me know:
https://github.com/Medium/medium-api-docs#32-publications

[jstoops]

Medium no longer allow you to create integration tokens/keys to use their API so had to use a different approach. I got it working using the user's RSS feed. I'm sure the content cleanup code could be better but this works to get me to the next stage.

In ETL config provide the user's RSS feed URL instead of individual articles, e.g. https://medium.com/feed/@pauliusztin

Then in the Medium.py crawler script loop through the items using feedparser, check the URL of the article hasn't been scraped yet, then create the entry and clean the content:

import feedparser
from bs4 import BeautifulSoup
from loguru import logger

from llm_engineering.domain.documents import ArticleDocument

from .base import BaseSeleniumCrawler


class MediumCrawler(BaseSeleniumCrawler):
    model = ArticleDocument

    def set_extra_driver_options(self, options) -> None:
        options.add_argument(r"--profile-directory=Profile 2")

    def extract(self, link: str, **kwargs) -> None:
        logger.info(f"Starting scrapping Medium article feed: {link}")
        substring_to_remove = "'value': '"

        # Parse the user's Medium RSS feed
        feed = feedparser.parse(link)

        # Loop through each entry (item) in the RSS feed
        for entry in feed.entries:
            old_model = self.model.find(link=entry.link)
            if old_model is not None:
                logger.info(f"Article already exists in the database: {entry.link}")
            else:
                title = str(entry.title)
                soup = BeautifulSoup(str(entry.content), "html.parser")
                serialized_entry = soup.get_text()
                index = serialized_entry.find(substring_to_remove)
                if index != -1:
                    content = serialized_entry[index + len(substring_to_remove):][:-3]
                else:
                    content = serialized_entry
                
                data = {
                    "Title": title if title else None,
                    "Content": content,
                }

                user = kwargs["user"]
                instance = self.model(
                    platform="medium",
                    content=data,
                    link=link,
                    author_id=user.id,
                    author_full_name=user.full_name,
                )
                instance.save()

                logger.info(f"Successfully scraped and saved article: {entry.link}")