🕸️

Common Crawl Explorer

CDX Index API · Amazon Open Data · Up to 1,000,000 records · Qualitative & NLP Analysis

🔍 Search Parameters

📄 Results

🧪 Analyse:
0 rows selected
#URLDomainCrawl Date MIMEStatusLangSizeDigestWARC FileCodes
Export:

🐍 Python Backend Script

Why use this?

The Common Crawl CDX API occasionally returns 504 / CORS errors when called directly from a browser, especially for wildcard queries or large result sets. Running this script locally bypasses CORS entirely and handles pagination robustly. Save the output JSON, then drag-and-drop or paste it into the interface via "Load JSON" below.

Script (Python 3.8+)

Loading...

Load local JSON result

🏷️ Qualitative Coding

Add Code

Memo

Export for QualCoder / NVivo

import json coded = json.load(open("commoncrawl_export.json")) codes = [r.get("codes",[]) for r in coded] print("Coded records ready for QualCoder import.")

🤖 BERTopic Modelling

About BERTopic

BERTopic uses BERT sentence embeddings + UMAP + HDBSCAN to produce coherent, human-readable topics. It outperforms LDA on short texts and social media data, preserving semantic meaning rather than relying on raw word counts.

Configuration & Python Script

pip install bertopic sentence-transformers umap-learn hdbscan from bertopic import BERTopic from sentence_transformers import SentenceTransformer import pandas as pd df = pd.read_csv("commoncrawl_export.csv") docs = df["url"].tolist() # replace with WET fetched text model = SentenceTransformer("all-MiniLM-L6-v2") topic_model = BERTopic(embedding_model=model, min_topic_size=10, nr_topics="auto", language="english") topics, probs = topic_model.fit_transform(docs) df["topic_id"] = topics print(topic_model.get_topic_info()) topic_model.visualize_barchart(top_n_topics=15).show() topic_model.save("berttopic_model")

📊 LDA Topic Modelling

About LDA

Latent Dirichlet Allocation treats each document as a mixture of latent topics. Best for longer text corpora. Pair with pyLDAvis for interactive topic exploration.

Python Script

pip install gensim pyLDAvis nltk import gensim, pyLDAvis.gensim_models, pandas as pd from gensim import corpora from nltk.corpus import stopwords df = pd.read_csv("commoncrawl_export.csv") stop = set(stopwords.words("english")) texts = [[w for w in d.lower().split() if w not in stop] for d in df["url"].tolist()] dct = corpora.Dictionary(texts) dct.filter_extremes(no_below=5, no_above=0.95) corpus = [dct.doc2bow(t) for t in texts] lda = gensim.models.LdaModel(corpus, id2word=dct, num_topics=20, passes=50, alpha="auto") vis = pyLDAvis.gensim_models.prepare(lda, corpus, dct) pyLDAvis.save_html(vis, "lda_topics.html")

💬 Context-Aware Sentiment

About

Context-aware sentiment uses transformer models (RoBERTa, DeBERTa) that understand negation, sarcasm, and nuanced language. Outputs: Positive / Negative / Neutral + intensity score (0–1).

Python Script

pip install transformers torch pandas tqdm from transformers import pipeline import pandas as pd from tqdm import tqdm df = pd.read_csv("commoncrawl_export.csv") texts = df["url"].tolist() # swap with WET text pipe = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment", truncation=True, max_length=512, batch_size=32) results = [] for i in tqdm(range(0, len(texts), 32)): results.extend(pipe(texts[i:i+32])) df["sentiment"] = [r["label"] for r in results] df["score"] = [r["score"] for r in results] df.to_csv("sentiment_results.csv", index=False) print(df["sentiment"].value_counts())