gorschal/wanish

About

Wanish is a lightweight Python package for extracting key information from web articles and generating concise summaries. It reduces full-length articles to a few essential sentences while preserving the core meaning.

.. note:: This project is in maintenance mode. No new features are being added, but the code remains stable and fully functional for production use.

Key Features

*   **Smart Summarization:** Extracts the main idea of the text into a short summary.
*   **Content Extraction:** Cleans HTML by removing ads, navigation, headers, and footers.
*   **Metadata Retrieval:** Automatically detects:
    *   Canonical URL
    *   Article Title
    *   Main Image URL
    *   Language Code (e.g., 'en', 'de', 'es')
*   **Clean HTML Output:** Returns structured, clean HTML based on schema.org data.

Installation
------------

You can install Wanish via pip:

.. code-block:: bash

    pip install wanish

Usage
-----

Basic usage example:

.. code-block:: python

    from wanish import Wanish

    # Initialize the parser
    wanish = Wanish()
    
    # Process the URL
    wanish.perform_url("https://example.com/article")

    # Access extracted data
    print(wanish.url)           # Canonical URL
    print(wanish.title)         # Article title
    print(wanish.image_url)     # Main image URL
    print(wanish.language)      # Language code (e.g., 'en')
    print(wanish.clean_html)    # Cleaned HTML content
    print(wanish.description)   # Generated summary (5 sentences by default)

Configuration
-------------

You can customize the behavior by passing optional arguments to the ``Wanish`` class:

.. code-block:: python

    wanish = Wanish(
        url="https://example.com/article",  # Optional: auto-processes URL on init
        positive_keywords=["main", "story"], # Class/ID patterns to keep
        negative_keywords=["banner", "ad"],  # Class/ID patterns to remove
        summary_sentences_qty=5,             # Length of summary
        headers={'user-agent': 'my-bot/1.0'} # Custom HTTP headers
    )

Parameter Details

• url (str, optional): If provided, the URL is processed immediately upon initialization. Default: None.
• positive_keywords (list, optional): List of CSS class or ID substrings that indicate important content (e.g., ["article-body", "content"]). Default: None.
• negative_keywords (list, optional): List of CSS class or ID substrings to exclude (e.g., ["sidebar", "ads", "footer"]). Default: None.
• summary_sentences_qty (int, optional): Number of sentences in the generated summary. Default: 5.
• headers (dict, optional): Custom HTTP headers for the request. Useful for bypassing basic bot protection. Default: None.

Special Thanks

This project builds upon the excellent work of the following libraries:

• NLTK <https://github.com/nltk/nltk>_
• python-readability <https://github.com/buriy/python-readability>_
• langid.py <https://github.com/saffsd/langid.py>_