Web Scraper API - Python Usage Guide

Overview

This document demonstrates how to interact with the Web Scraper API using Python. You will learn how to: - Retrieve data (GET) - Insert new records (POST) - Apply filters, pagination, and sorting - Handle conflicts when inserting data - Batch insert multiple records efficiently

Authentication

All requests require an api-key in the headers:

headers = {
    "Content-Type": "application/json",
    "api-key": "YOUR_API_KEY"
}

---

1️⃣ Retrieving Data (GET Requests)

Capitol Trades Data

Get all trade tickers

url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker"
response = requests.get(url, headers=headers)
print(response.json())

Get trades by a specific politician

url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=politician,trade_ticker&filter=politician=John Doe"
response = requests.get(url, headers=headers)
print(response.json())

Get trades within a date range

url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker,traded_date&date_from=2024-01-01&date_to=2024-03-10"
response = requests.get(url, headers=headers)
print(response.json())

Get trades by type (BUY or SELL)

url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker,type&filter=type=BUY"
response = requests.get(url, headers=headers)
print(response.json())

Get stock tickers

url = "https://srsimpson.it.com/capitol_trades_api/get_stock_tickers?fields=ticker,name"
response = requests.get(url, headers=headers)
print(response.json())

Get stock prices for a specific ticker

url = "https://srsimpson.it.com/capitol_trades_api/get_stock_prices?fields=ticker,price,timestamp&filter=ticker=AAPL"
response = requests.get(url, headers=headers)
print(response.json())

Get latest 5 crypto prices

url = "https://srsimpson.it.com/capitol_trades_api/get_crypto_prices?fields=ticker,price,timestamp&limit=5&sort=timestamp DESC"
response = requests.get(url, headers=headers)
print(response.json())

---

2️⃣ Inserting Data (POST Requests)

Insert a single trade record

data = [
    {
        "politician": "John Doe",
        "party": "Independent",
        "chamber": "House",
        "trade_ticker": "AAPL",
        "published_day": "01",
        "published_year": "2024",
        "published_date": "2024-03-01",
        "traded_day": "01",
        "traded_year": "2024",
        "traded_date": "2024-03-01",
        "filed_after": 10,
        "owner": "Self",
        "type": "BUY",
        "size": "Large",
        "price": "150.00"
    }
]
url = "https://srsimpson.it.com/capitol_trades_api/insert_capitol_trades"
response = requests.post(url, json=data, headers=headers)
print(response.json())

Batch Insert Multiple Stock Prices

data = [
    {
        "ticker": "AAPL",
        "name": "Apple Inc.",
        "price": 150.25,
        "timestamp": "2024-03-10T14:30:00"
    },
    {
        "ticker": "GOOGL",
        "name": "Alphabet Inc.",
        "price": 2750.50,
        "timestamp": "2024-03-10T14:30:00"
    }
]
url = "https://srsimpson.it.com/capitol_trades_api/insert_stock_prices"
response = requests.post(url, json=data, headers=headers)
print(response.json())

---

3️⃣ Full Example Script for Read-Only and Read-Write API Keys

import requests

readonly_headers = {
    "Content-Type": "application/json",
    "api-key": "YOUR_READONLY_API_KEY"
}

readwrite_headers = {
    "Content-Type": "application/json",
    "api-key": "YOUR_READWRITE_API_KEY"
}

# GET Request with Read-Only API Key
print("Fetching trade tickers (Read-Only API Key)...")
url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker"
response = requests.get(url, headers=readonly_headers)
print(response.json())

# Attempting to Insert with Read-Only API Key (Should Fail)
print("Attempting to insert a trade record (Read-Only API Key)...")
data = [{
    "politician": "Jane Doe",
    "party": "Democrat",
    "chamber": "Senate",
    "trade_ticker": "TSLA",
    "published_date": "2024-03-01",
    "traded_date": "2024-03-01",
    "filed_after": 10,
    "owner": "Spouse",
    "type": "SELL",
    "size": "Medium",
    "price": "700.00"
}]
response = requests.post(url, json=data, headers=readonly_headers)
print(response.json())  # Should return an error

# Insert with Read-Write API Key (Should Succeed)
print("Inserting trade record (Read-Write API Key)...")
response = requests.post(url, json=data, headers=readwrite_headers)
print(response.json())  # Should return success message

Expected Output

Fetching trade tickers (Read-Only API Key)...
{"capitol_trades": [{"trade_ticker": "AAPL"}, {"trade_ticker": "TSLA"}]}

Attempting to insert a trade record (Read-Only API Key)...
{"detail": "Forbidden: Write access required"}

Inserting trade record (Read-Write API Key)...
{"message": "Inserted 1 records successfully in capitol_trades."}

🚀 Now you're ready to test your API with different access levels!

---

3️⃣ Creating Tables (Admin Only)

Create a New Table

import requests

url = "https://srsimpson.it.com/capitol_trades_api/admin_create_table"
headers = {
    "Content-Type": "application/json",
    "api-key": "YOUR_ADMIN_API_KEY"
}

data = {
    "table_name": "test_table",
    "columns": [
        {"name": "id", "type": "INTEGER"},
        {"name": "name", "type": "TEXT"},
        {"name": "created_at", "type": "TIMESTAMP"}
    ]
}

response = requests.post(url, json=data, headers=headers)
print(response.json())

📌 Expected Responses

✅ Success (if table is created):

{
    "message": "Table `test_table` created successfully."
}

🚫 Failure (if permissions are missing):

{
    "detail": "Forbidden: Admin access required"
}

🚫 Failure (if table already exists):

{
    "detail": "Table `test_table` already exists."
}

🚀 Now, admins can dynamically create tables in the database!