A Chrome app to rule them all (Paywalls) : Part 1

They've been telling us internet will make us free, that we will have the knowledge at the reach of our browser, infinite possibilities and all that stuff… until you hit the (pay)wall:

"Paywall

When you find this message, you have 2 options, you pay the member fee or you can use all your knowledge to bypass it¹

Let's go that way. There are various types of paywall control techniques:

By cookie (we will focus on this one today)
By redirecting to another site
By hiding content using JavaScript

Medium use cookies to track us, so the first option would be to delete the cookie that triggers this paywall (field sid in the medium cookie by the way). The problem is I would have to login again then, because maybe after reading it i want to bookmark it o follow the author (my problem is with the paywall not the content). But why bother, just open the article in private/incognito mode and voilà, no paywall at all.

"No Paywall

So how can we achieve this with a less friction? Let's say we build a Chrome extension that can detect we were stopped by the paywall and automatically open the site in private mode, pretty simple no? If you want a quick guide in how to make a default extension see this Google's guide, i will focus on the paywall bypass, download the template app and let's start with that.

The extension

Our app consist of mainly of :

manifest.json holds the permissions and declare the other files
background.js for communicating the extension
hidden.js detects the paywall warning and triggers the private window
display.js we will use it after in this series

Let's dive into manifest:

{
  "name": "Gandalf",
  "version": "1.0",
  "description": "You shall not redirect",
  "manifest_version": 2,
  "content_security_policy": "script-src 'self' https://ajax.googleapis.com; object-src 'self'",
  "background": {
   "scripts": ["app/background.js"] <---- 1
  },
  "permissions": [ <---- 2
    "webRequest",
    "webRequestBlocking",
    "tabs",
    "<all_urls>"
  ],
  "browser_action": { <---- 3 
      "default_title": "You shall not pass",   
      "default_popup": "popup.html"
  },
  "icons": {
    "16": "logo-small.png",
    "48": "logo-small.png",
    "128": "logo-small.png"
  },
  "content_scripts": [
    {
      "matches": ["*://*.uy/*"],
      "js": ["app/display.js"] 
    },
    {
      "matches": ["<all_urls>"],
      "js": ["app/hidden.js"] <---- 4
    }
  ]
}

Link to the script that holds the extension logic, add listeners and define some constants.
The extension need some permissions, webRequest and webRequestBlocking for blocking a request (for the next chapter), tabs for creating a new window or tab. <all_urls> let us operate in all websites, you can change this if you only want specific sites.
Just a title for our extension and popup.html is what we render when someone click on the extension icon.
Content scripts can change the site content, this one apply to all urls and invoke hidden.js.

So we have defined that for all urls, hidden.js will be executed. This script can access the site body, check if paywall is present and send a message (internal messaging is the way our content scripts communicates with the extension).

var paywallId = ""; // We need an id to know if paywall is shown
window.onload = function() { // When site loads this will be called
  var content = document.getElementById(paywallId); // Find paywall
  if (content != null){
    // Send a message to the background with current url
    chrome.runtime.sendMessage({
      incognito: true, 
      url: window.location.href
      }, 
      function(response) {
        console.log("Opening page in incognito", response);
      }
    );
  }
}

For medium we can use the id paywall-background-color to check if we have to open in incognito; now its time to check the background script:

window.chrome.runtime.onMessage.addListener(
  function(request, sender, sendResponse) {
    // Check if the message is ours
    if (request.incognito == true){
      // Create incognito window with the requested url
      openIncognito = chrome.windows.create({"url": request.url, "incognito": request.incognito});
      sendResponse({msg: "Time to read"});
    }
  });

Now that we have our extension ready, we can load it to the browser and start reading non-stop. Next in the series we will try to beat the redirection type of paywalls.

Resources

All the extension is doing is removing some friction from the user (that could always copy the link, enter in private mode and paste it), by no means we are hacking or altering medium behaviour. ↩

A Chrome app to rule them all (Paywalls) : Part 1

Paywall by cookie

The extension

Resources

Footnotes