Web Scraper API - C# Usage Guide

Overview

This document demonstrates how to interact with the Web Scraper API using C# and HttpClient. You will learn how to: - Retrieve data (GET) - Insert new records (POST) - Apply filters, pagination, and sorting - Handle conflicts when inserting data - Batch insert multiple records efficiently

Authentication

All requests require an api-key in the headers:

httpClient.DefaultRequestHeaders.Add("api-key", "YOUR_API_KEY");

---

1️⃣ Retrieving Data (GET Requests)

Get all trade tickers

string url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker";
HttpResponseMessage response = await client.GetAsync(url);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Get trades by a specific politician

string url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=politician,trade_ticker&filter=politician=John Doe";
HttpResponseMessage response = await client.GetAsync(url);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Get stock tickers

string url = "https://srsimpson.it.com/capitol_trades_api/get_stock_tickers?fields=ticker,name";
HttpResponseMessage response = await client.GetAsync(url);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Get stock prices for a specific ticker

string url = "https://srsimpson.it.com/capitol_trades_api/get_stock_prices?fields=ticker,price,timestamp&filter=ticker=AAPL";
HttpResponseMessage response = await client.GetAsync(url);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Get latest 5 crypto prices

string url = "https://srsimpson.it.com/capitol_trades_api/get_crypto_prices?fields=ticker,price,timestamp&limit=5&sort=timestamp DESC";
HttpResponseMessage response = await client.GetAsync(url);
Console.WriteLine(await response.Content.ReadAsStringAsync());

---

2️⃣ Inserting Data (POST Requests)

Insert a single trade record

string url = "https://srsimpson.it.com/capitol_trades_api/insert_capitol_trades";
string json = "[{ \"politician\": \"John Doe\", \"party\": \"Independent\", \"chamber\": \"House\", \"trade_ticker\": \"AAPL\", \"published_date\": \"2024-03-01\", \"traded_date\": \"2024-03-01\", \"filed_after\": 10, \"owner\": \"Self\", \"type\": \"BUY\", \"size\": \"Large\", \"price\": \"150.00\" }]";

HttpContent content = new StringContent(json, Encoding.UTF8, "application/json");
HttpResponseMessage response = await client.PostAsync(url, content);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Batch Insert Multiple Stock Prices

string url = "https://srsimpson.it.com/capitol_trades_api/insert_stock_prices";
string json = "[ { \"ticker\": \"AAPL\", \"name\": \"Apple Inc.\", \"price\": 150.25, \"timestamp\": \"2024-03-10T14:30:00\" }, { \"ticker\": \"GOOGL\", \"name\": \"Alphabet Inc.\", \"price\": 2750.50, \"timestamp\": \"2024-03-10T14:30:00\" } ]";

HttpContent content = new StringContent(json, Encoding.UTF8, "application/json");
HttpResponseMessage response = await client.PostAsync(url, content);
Console.WriteLine(await response.Content.ReadAsStringAsync());

---

3️⃣ Full Example Script for Read-Only and Read-Write API Keys

using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        HttpClient client = new HttpClient();

        HttpRequestMessage readonlyRequest = new HttpRequestMessage();
        readonlyRequest.Headers.Add("api-key", "YOUR_READONLY_API_KEY");

        HttpRequestMessage readwriteRequest = new HttpRequestMessage();
        readwriteRequest.Headers.Add("api-key", "YOUR_READWRITE_API_KEY");

        // GET Request with Read-Only API Key
        Console.WriteLine("Fetching trade tickers (Read-Only API Key)...");
        string url = "https://srsimpson.it.com/capitol_trades_api/get_capitol_trades?fields=trade_ticker";
        HttpResponseMessage response = await client.GetAsync(url);
        Console.WriteLine(await response.Content.ReadAsStringAsync());

        // Attempting to Insert with Read-Only API Key (Should Fail)
        Console.WriteLine("Attempting to insert a trade record (Read-Only API Key)...");
        string json = "[{ \"politician\": \"Jane Doe\", \"party\": \"Democrat\", \"chamber\": \"Senate\", \"trade_ticker\": \"TSLA\", \"published_date\": \"2024-03-01\", \"traded_date\": \"2024-03-01\", \"filed_after\": 10, \"owner\": \"Spouse\", \"type\": \"SELL\", \"size\": \"Medium\", \"price\": \"700.00\" }]";

        HttpContent content = new StringContent(json, Encoding.UTF8, "application/json");
        response = await client.PostAsync(url, content);
        Console.WriteLine(await response.Content.ReadAsStringAsync());

        // Insert with Read-Write API Key (Should Succeed)
        Console.WriteLine("Inserting trade record (Read-Write API Key)...");
        client.DefaultRequestHeaders.Clear();
        client.DefaultRequestHeaders.Add("api-key", "YOUR_READWRITE_API_KEY");
        response = await client.PostAsync(url, content);
        Console.WriteLine(await response.Content.ReadAsStringAsync());
    }
}

Expected Output

Fetching trade tickers (Read-Only API Key)...
{"capitol_trades": [{"trade_ticker": "AAPL"}, {"trade_ticker": "TSLA"}]}

Attempting to insert a trade record (Read-Only API Key)...
{"detail": "Forbidden: Write access required"}

Inserting trade record (Read-Write API Key)...
{"message": "Inserted 1 records successfully in capitol_trades."}

🚀 Now you're ready to test your API with different access levels!