Skip to content

hsnamr/FediListener

BranchesTags

Repository files navigation

ActivityPub Listener

A REST API service for social listening and monitoring of ActivityPub/Fediverse instances. Monitor and analyze content from federated social networks such as Friendica, Mastodon, Lemmy, PeerTube, GNU social, Pleroma and other ActivityPub-compatible platforms.

🎯 Overview

ActivityPub Listener is a Spring Boot-based service that enables organizations to:

  • Monitor keywords, accounts, instances, and regions across Fediverse instances
  • Collect and analyze ActivityPub activities in real-time
  • Retrieve analytics and insights from monitored content
  • Manage monitoring configurations via REST API

Target Platforms: ActivityPub/Fediverse instances only (Mastodon, Lemmy, PeerTube, etc.)
Excluded Platforms: Commercial social networks (Twitter/X, Facebook, Instagram, LinkedIn, TikTok)

✨ Features

Monitor Management

  • CRUD Operations: Create, read, update, and delete monitoring configurations
  • Multiple Monitor Types:
    • Keyword-based monitoring
    • Account/profile analysis
    • Regional/geographic monitoring
    • Managed accounts and pages
  • Lifecycle Management: Approval workflow, pause/resume functionality
  • Multi-Instance Support: Monitor across multiple Fediverse servers simultaneously

ActivityPub Integration

  • Protocol Compliance: Full W3C ActivityPub 1.0 specification support
  • Federation: Connect to and collect data from any ActivityPub-compatible instance
  • Activity Collection: Monitor Create, Update, Delete, Announce, Like, and other activities
  • Actor Tracking: Track users, groups, and organizations across instances
  • WebFinger Discovery: Automatic actor and instance discovery

Data & Analytics

  • Real-time Streaming: Kafka-based real-time data streaming
  • Social Listening Data: Retrieve analytics and metrics for monitors
  • Widget Support: Various data visualization widgets (engagements, posts, topics, sentiment, etc.)
  • Advanced Filtering: Filter by topics, sentiment, languages, accounts, and more
  • Date Range Queries: Flexible date range filtering

Technical Features

  • RESTful API: Comprehensive REST API with OpenAPI documentation
  • Message Queue Integration: Kafka for async processing and real-time updates
  • Database: MongoDB for persistent storage
  • Authentication: JWT-based authentication and authorization
  • Rate Limiting: Per-instance rate limiting with exponential backoff
  • Health Monitoring: Instance health checks and monitoring

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Client Applications                      │
└────────────────────┬────────────────────────────────────────┘
                     │ HTTP/REST API
                     │
┌────────────────────▼────────────────────────────────────────┐
│         ActivityPub Listener Service (Spring Boot)          │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  REST API Controllers                                 │  │
│  │  Business Logic Layer                                 │  │
│  │  ActivityPub Integration Layer                        │  │
│  │  Data Access Layer (JPA)                              │  │
│  └──────────────────────────────────────────────────────┘  │
└──────┬───────────────────┬───────────────────┬──────────────┘
       │                   │                   │
┌──────▼──────┐   ┌────────▼────────┐  ┌──────▼──────────────┐
│  Database   │   │  Message Queue  │  │  Fediverse Instances │
│  (MongoDB)   │   │    (Kafka)       │  │  - mastodon.social   │
│             │   │                 │  │  - lemmy.ml          │
│             │   │                 │  │  - peertube.io       │
└─────────────┘   └─────────────────┘  └─────────────────────┘

🛠️ Technology Stack

Core Framework

  • Java: 17 or higher (LTS)
  • Spring Boot: 3.2.x
  • Spring Web: REST API
  • Spring Data JPA: Database access
  • Spring Kafka: Kafka integration

ActivityPub Libraries

  • activityPub4j: W3C ActivityPub implementation for Java/Spring Boot
  • Mastodon4J: Mastodon API client library
  • BigBone: Alternative Mastodon client (if needed)

Database & Messaging

  • MongoDB: 7.0+ (primary database)
  • Apache Kafka: 3.5+ (message queue)

Additional Dependencies

  • Jackson: JSON processing
  • Bean Validation: Input validation
  • Lombok: Reduce boilerplate
  • JWT: Authentication
  • JUnit 5 + Mockito: Testing

🚀 Quick Start

Prerequisites

  • Java 17 or higher
  • Maven 3.8+ or Gradle 7+
  • MongoDB 7.0+
  • Apache Kafka 3.5+ (with Zookeeper)
  • Docker & Docker Compose (optional, for local development)

Local Development Setup

  1. Clone the repository

    git clone <repository-url>
cd ActivityPubListener

  2. Set up environment variables

    cp .env.example .env
# Edit .env with your configuration

  3. Start dependencies with Docker Compose

    docker-compose up -d

    This starts MongoDB, Zookeeper, and Kafka containers.

  4. Build and run the application

    ./mvnw spring-boot:run
# or
./gradlew bootRun

  5. Verify the service is running

    curl -H "API-Version: v1" http://localhost:8080/api/actuator/health

Configuration

Key environment variables:

# Database
MONGODB_URI=mongodb://localhost:27017/activitypub_listener
MONGODB_DATABASE=activitypub_listener

# Kafka
KAFKA_BOOTSTRAP_SERVERS=localhost:9092
KAFKA_CONSUMER_GROUP_ID=activitypub-listener

# ActivityPub
ACTIVITYPUB_USER_AGENT=ActivityPubListener/1.0
ACTIVITYPUB_DEFAULT_RATE_LIMIT=300

# JWT
JWT_PUBLIC_KEY=your_jwt_public_key
JWT_ISSUER=AMRITECH

📡 API Overview

Base URL

http://localhost:8080/api

API Versioning

All endpoints use header-based versioning. Include the API-Version header in your requests:

API-Version: v1

If no version header is provided, the API defaults to v1.

Authentication

All endpoints require JWT authentication:

Authorization: Bearer <jwt_token>

Monitor Management

Create Monitor

POST /api/monitors
API-Version: v1
Content-Type: application/json

{
  "name": "Brand Monitoring",
  "monitor_type_id": 1,
  "product_id": 1,
  "data_sources": [1],
  "monitor_options": {
    "KEYWORD": {
      "MASTODON": {
        "keywords": "brand, product",
        "spam_keywords": "spam, test"
      }
    }
  },
  "languages": "en,ar",
  "exact_search": false
}

List Monitors

GET /api/monitors?page=1&per_page=10&search=brand
API-Version: v1

Get Monitor

GET /api/monitors/{id}
API-Version: v1

Update Monitor

PUT /api/monitors/{id}
API-Version: v1
Content-Type: application/json

Delete Monitor

DELETE /api/monitors/{id}
API-Version: v1

Pause/Resume Monitor

POST /api/monitors/{id}/pause
API-Version: v1

POST /api/monitors/{id}/resume
API-Version: v1

Social Listening Data

Request Social Listening Data

POST /api/social-listening/data
API-Version: v1
Content-Type: application/json

{
  "monitor_id": 123,
  "data_source": "MASTODON",
  "page_name": "account_page",
  "start_date": 1706352000,
  "end_date": 1706438400,
  "filters": {
    "topics": [1, 2],
    "sentiment": ["positive"],
    "languages": ["en"]
  },
  "widgets_names": ["engagements", "posts", "topics"],
  "page_number": 1
}

Get Available Widgets

GET /api/social-listening/widgets?monitor_id=123&data_source=MASTODON&page_name=account_page
API-Version: v1

ActivityPub-Specific Endpoints

List Fediverse Instances

GET /api/instances
API-Version: v1

Discover Actor

GET /api/actors/discover?resource=acct:[email protected]
API-Version: v1

POST /api/actors/discover?resource=... is also supported for backward compatibility.

Get Actor

GET /api/actors/{actor_id}
API-Version: v1

Get Actor Activities

GET /api/actors/{actor_id}/activities?page=1&per_page=20
API-Version: v1

Trigger Outbox Poll (collect new activities)

POST /api/actors/{actor_id}/poll?monitorId=optional
API-Version: v1

List Fediverse Instances

GET /api/instances
API-Version: v1

Optional query: ?paged=true&page=1&per_page=20

Instance Health Check

GET /api/instances/{id}/health
API-Version: v1

📊 Monitor Types

1. Keyword Monitor

Monitor content containing specific keywords across Fediverse instances.

Use Cases:

  • Brand mention tracking
  • Topic monitoring
  • Trend analysis

2. Account Monitor

Monitor specific users and their activities.

Use Cases:

  • Competitor monitoring
  • Influencer tracking
  • Community analysis

3. Instance Monitors

4. Regional Monitor

Monitor content based on geographic regions (if location data is available).

Use Cases:

  • Geographic trend analysis
  • Regional sentiment monitoring

🔧 Development

Project Structure

src/
├── main/
│   ├── java/
│   │   └── com/activitypub/listener/
│   │       ├── ActivityPubListenerApplication.java
│   │       ├── config/
│   │       ├── controller/
│   │       ├── service/
│   │       ├── repository/
│   │       ├── model/
│   │       ├── dto/
│   │       ├── exception/
│   │       └── activitypub/
│   └── resources/
│       ├── application.properties
│       └── db/migration/
└── test/

Running Tests

./mvnw test
# or
./gradlew test

Code Style

  • Follow Java coding conventions
  • Use Lombok to reduce boilerplate
  • Write comprehensive unit and integration tests

External Resources

🤝 Contributing

Contributions are welcome! Please follow these steps:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add some amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Guidelines

  • Write tests for new features
  • Follow the existing code style
  • Update documentation as needed
  • Ensure all tests pass before submitting

📝 License

See LICENSE file for details.

🐛 Known Issues & Limitations

  • Rate Limiting: Each Fediverse instance has different rate limits. The service implements per-instance rate limiting, but aggressive monitoring may hit limits.
  • Content Format: ActivityPub content may be in HTML or Markdown. Content extraction and normalization may be needed.
  • Federation Delays: Activities may arrive with delays due to federation. The service handles out-of-order activities.
  • Instance Diversity: Different ActivityPub implementations (Mastodon, Lemmy, PeerTube) may require different handling.

🗺️ Roadmap

  • Project specification and planning
  • Phase 1: Foundation & Core Infrastructure ✅
    • Spring Boot project setup with Maven
    • Database schema with MongoDB collections
    • JPA entities and repositories (9 entities, 7 repositories)
    • REST API for monitor management (7 endpoints)
    • ActivityPub client integration (WebFinger, actor discovery)
    • Docker Compose with MongoDB and Kafka
    • Error handling and validation
  • Phase 2: ActivityPub Data Collection ✅
    • ActivityPub polling implementation (outbox, pagination, ActivityStreams parser)
    • Data collection (OutboxPollingService, CollectedActivity, Kafka activities topic)
    • Kafka integration for activity streaming and tracker config (tracker-new)
    • Instance management (list, CRUD, health check)
    • API alignment: GET /api/actors/discover, GET /api/actors/{id}/activities, POST /api/actors/{id}/poll
  • Phase 3: Monitor Types & Business Logic ✅
    • Keyword monitor (matching + spam filtering via ActivityFilterService)
    • Account analysis monitor (follow/excluded accounts)
    • Regional monitor (MBR placeholder; ActivityFilterService)
    • Monitor lifecycle (approve/reject endpoints, Kafka monitor-lifecycle)
    • List monitors: filters (monitor_type_id, product_id, data_source_id, is_approved, paused) and sort (id, name, created_at, monitor_type, status, etc.)
    • MonitorDTO: companyId, dataSources from all monitor types
    • ManagedAccount/ManagedPage support in create/update
    • GET /api/monitors/{id}/activities
  • Phase 4: Social Listening Data & Analytics ✅
    • POST /api/social-listening/data (validation, Kafka send, topic + consumer_group response)
    • GET /api/social-listening/widgets?monitor_id=&data_source=&page_name=
    • ParameterPreparationService + SocialListeningAnalyticsMessage (filters as JSON)
    • Public API async: POST/GET /api/public-api/social-listening/data (job store)
    • Filters support (topics, sentiment, languages, etc.) in request and Kafka payload
  • Phase 5: Advanced Features (ActivityPub Server)
    • ActivityPub server implementation
    • Enhanced federation
    • Performance optimization
    • Monitoring & observability
  • Phase 6: Testing & Documentation
    • Unit and integration tests
    • API documentation (OpenAPI/Swagger)
    • Deployment guides

See IMPLEMENTATION_PLAN.md for detailed roadmap.

📞 Support

For issues, questions, or contributions:

  • Open an issue on GitHub
  • Check the documentation in SPECIFICATION.md and IMPLEMENTATION_PLAN.md

🙏 Acknowledgments

  • activityPub4j - ActivityPub Java library
  • Mastodon4J - Mastodon API client
  • The Fediverse community for the ActivityPub protocol

Built with ❤️ for the Fediverse

About

A listener and monitor for ActivityPub instances such as Friendica, Lemmy, Mastodon, PeerTube and others.

Resources

Readme

License

AGPL-3.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages