Today¶
- What kind of link data does Twitter provide?
- How to extract link data from Tweets (
urlexpander.tweet_utils.get_link) - Processing data by expanding shortened URLs (
urlexpander.expand) - Avenues of analysis with link data (
urlexpander.tweet_utils.count_matrix,urlexpander.html_utils.get_webpage_meta) - Using links as features to predict political affiliation
Software for this tutorial is found in this requirements.txt file, and can be downloaded as follows:
pip install -r requirements.txtDownload data here:
python download_data.pyNOTE: at the time of this writing, download_data.py does not work! Please go to OSF in the meantime.
What Kind of Link Data Does Twitter Provide?¶
If you have yet to look at the backend of a Tweet here you go: https://bit.ly/tweet_anatomy_link.
Today I'll will show you how to extract and work with links from Tweets. We're working with Tweets from members of congress collected by Greg Eady.
import os
import json
import glob
import itertools
from multiprocessing import Pool
from tqdm import tqdm
import pandas as pd
import urlexpander
from smappdragon import JsonCollection
from config import INTERMEDIATE_DIRECTORY, \
RAW_TWEETS_DIRECTORY, \
CONGRESS_METADATA_DIRECTORY
Let's preview one file. The file is saved as a newline-delimited json file like this
{"tweet_id" : "123", "more_data" : {"here" : "it is"}
{"tweet_id" : "124", "more_data" : {"here" : "it is again"}and bzip2 compressed!
f
The file structure is identical to how we store all Tweet data at SMaPP. We developed software like smappdragon to work with Tweets:
collect = JsonCollection(f, compression='bz2',
throw_error=False,
verbose=1)
smappdragon's JsonCollection class reads through JSON files as a generator.
collect
The con is that generators are hard to interpret, the pro is that they don't store any data in memory. We access the data on a row-by-row basis by iterating through the collect object. Here we only get the first row, we can see the contents by printing row:
for row in collect.get_iterator():
print(json.dumps(row, indent=1)[:100])
break
How do we get the links?¶
Each Tweet can have more than one link, thus we need to unpack those values! We created the . urlexpander.tweet_utils.get_link() has a function to do just this...
Once again we have another generator
# returns a genrator, which is uninterpretable!
urlexpander.tweet_utils.get_link(row)
# we can access the data by iterating through it.
for link_meta in urlexpander.tweet_utils.get_link(row):
print(link_meta)
def read_file_extract_links(f):
'''
This function takes in a Tweet file that bzip2-compressed,
newline-deliminted json, and returns a list of dictionaries
for link data.
'''
# read the json file into a generator
collection = JsonCollection(f, compression='bz2', throw_error=False)
# iterate through the json file, extract links, flatten the generator of links
# into a list, and store into a Pandas dataframe
df_ = pd.DataFrame(list(
itertools.chain.from_iterable(
[urlexpander.tweet_utils.get_link(t)
for t in collection.get_iterator()
if t]
)))
df_['file'] = f
return df_.to_dict(orient='records')
We can iterate through files and run the function iteratively. From there, we can instantiate a Pandas DataFrame.
data = []
for f in tqdm(files[:2]):
# read the json file into a generator
data.extend(read_file_extract_links(f)
df_links = pd.DataFrame(data)
Advanced (but practical) usage¶
The for loop is is slow.. Also this task is not memory intensive.
Parallelize the task using the Pool class from the Mulitprocessing package.
Each core on our computer will read a JSON file of Tweets and filter for links.
data = []
with Pool(4) as pool:
iterable = pool.imap_unordered(read_file_extract_links, files)
for link_data in tqdm(iterable, total=len(files)):
data.extend(link_data)
# link meta -> dataframe
df_links = pd.DataFrame(data)
# save it
df_links.to_csv(file_raw_links, index=False)
# preview it
df_links.head(2)
How useful is this data?¶
The bulk of URLs we encounter in the wild are sent through a link shortener. For this dataset 30% of all URLs are from known link shorteners
len(df_links[df_links['link_domain'].isin(short_domains)]) \
/ len(df_links)
Link shorteners record transactional information whenever that link is clicked. Unfortunately it makes it hard for us to see what was being shared.
links = df_links['link_url_long'].tolist()
links[-5:]
This is why urlexpander was made. We can run the expand function on single URLs, as well as a list of URLs.
urlexpander.expand(links[-5])
By default, urlexpander will expand every URL it is shown. However you can pass a boolean function (one the returns True or False based on an inputted string) to the filter_function parameter.
urlexpander.expand(links[-5:], filter_function=urlexpander.is_short)
What's happening behind the scenes?¶
['abc.com/123', 'bbc.co.uk/123', 'abc.com/123', 'bit.ly/cbc23']
--> Remove duplicates
['abc.com/123', 'bbc.co.uk/123', 'bit.ly/cbc23']
--> Filter for shortened URLs (optional)
['bit.ly/cbs23']
--> Check the cache file (optional), Unshorten new urls
[{'original_url': 'bit.ly/cbs23', 'resolved' : 'cspan.com/123'}]
--> swap short URLs for full URLs
['abc.com/123', 'bbc.co.uk/123', 'abc.com/123', 'cspan.com/123']
urlexpander parallelizes, filters and caches the input, which is essential for social media data.
?urlexpander.expand
The above is a toy example with 5 links, let's see how this works for 1.7 Mil links. For reference, this took me an hour on HPC.
resolved_urls = urlexpander.expand(links,
filter_function=urlexpander.is_short,
n_workers=64,
chunksize=1280,
cache_file=file_cache,
verbose=1)
len(resolved_urls)
df_links['link_resolved'] = resolved_urls
df_links['link_resolved_domain'] = df_links['link_resolved'].apply(urlexpander.get_domain)
df_links.to_csv(file_expanded_links, index=False)
Analytics¶
With the links resolved, how can we use links as data?
df_links = pd.read_csv(file_expanded_links)
df_links.head(2)
We can get an overview of the most frequently shared domains:
df_links['link_resolved_domain'].value_counts().head(15)
Text-based URL Metadata¶
We can also look at the contents of each URL. Twitter provides URL metadata (if you pay), we provided a workaround!
url = 'http://www.rollcall.com/news/house_hydro_bill_tests_water_for_broad_energy_deals-224277-1.html'
meta = urlexpander.html_utils.get_webpage_meta(url)
meta
Categorizing what is shared¶
If we want to know more about what kinds of information are being shared by members of congress, we can enrich the dataset by joining metadata about domains. In this example we will use the Local News Dataset to inspect the local media outlets that members of congress share:
local_news_url = 'https://raw.githubusercontent.com/yinleon/LocalNewsDataset/master/data/local_news_dataset_2018.csv'
df_local = pd.read_csv(local_news_url)
df_local = df_local[~(df_local.domain.isnull()) &
(df_local.domain != 'facebook.com')]
df_local.head()
# this is a SQL-like join in Pandas that merges the two datasets based on domain name!
df_links_state = df_links.merge(df_local,
left_on='link_resolved_domain',
right_on='domain')
This dataset unlocks insights regarding the locality of news articles shared, as well as media ownership.
df_links_state.state.value_counts().head(20)
df_links_state.owner.value_counts().head(25)
df_links_state[df_links_state.owner == 'Sinclair']['user_id'].value_counts().head(10)
This is one example of a dataset enrichment, you can create your own categorizations and join them in. Alexa.com is a good starting place, and Greg and Andreu have the most experience categorizing websites.
User-Aggregated Acvitity¶
The frequency of domains shared per-user make rich features. We can aggregate the data using the following utility function:
count_matrix = urlexpander.tweet_utils.count_matrix(
df_links,
user_col='user_id',
domain_col='link_resolved_domain',
min_freq=20,
)
count_matrix.head()
Are links good features to predict political affiliation?¶
Let's see how the count_matrix features fair as predictors of political affiliation.
Unsupervised Learning¶
This will be an exploratory step, where we will try to visualze the dataset of counts of links shared by user. We will reduce the high-dimensional data (where each domain is one dimension) to two-dimensions for visualization using the UMAP algorithm (much like the populat TSNE algorithm).
count_matrix = urlexpander.tweet_utils.count_matrix(
df_links_,
user_col='user_id',
domain_col='link_resolved_domain',
min_freq=5,
exclude_domain_list=exclude,
)
count_matrix.head(2)
viz_umap_embed(count_matrix,
title="Link Sharing of Members of Congress Embedded by UMAP",
threshold=5, n_neighbors=50,
min_dist=0.1, metric='dice',
random_state=303)
Supervised Learning¶
In the unsupervised case, we already see Democrats and Republicans sectioned off. Here we will fit a logistic regression model to predict whether a congress member is a Democrat or a Republican based on the count_matrix we just created.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
# create the training set
X, y = count_matrix_.values, parties
X_train, X_test, y_train, y_test = train_test_split(X, y,
random_state=303,
test_size=.15)
len(X_train), len(X_test)
logreg = LogisticRegression(penalty='l2', C=.7,
solver='liblinear',
random_state=303)
logreg.fit(X_train, y_train)
logreg.score(X_test, y_test)
Evaluation¶
plot_confusion_matrix(cnf_matrix, classes=class_names,
title='Confusion matrix, without normalization')
Who did we get wrong?¶
df_meta.set_index('twitter_id').loc[
y_test[y_test != y_pred].index
][['twitter_name']]
How did we do with Independents?¶
df_ind = df_meta.set_index('twitter_id').loc[count_matrix_indep_.index][['twitter_name']]
df_ind['preds'] = logreg.predict(count_matrix_indep_)
df_ind
K-Fold Cross Validation¶
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))