Web Proxies

Background

A web proxy can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable. Web proxies work by forwarding requests to the original website, and providing a different URL to access that content. By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website’s URL.

Web proxies also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content. This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the deployment of new URLs, and for the distribution lists to allow applications and end-users to discover new URLs.

Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge.

Simple vs. Smart Proxies

We currently distinguish between simple and smart proxies. This distinction will become redundant in the future as we aim to allow for all providers to offer the smart proxy functionality. Smart proxies differ from simple proxies in that they perform active rewriting of the responses from the original website. This modifies links to resources to ensure that they are always loaded via the mirror system rather than from the original—blocked—website.

Smart proxies are only supported on AWS right now.

Architecture

../_images/architecture.png

Before configuring any web origins to create mirrors, you must first configure an origin group. The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration. The shared configuration will include which pools will require mirrors for the origins.

For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed. This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors. Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality. No smart proxy instance will be created for providers where it is not necessary.

Web origins represent the original website that has been blocked.

Web Origins

Rate Limiting

CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure that the network resources are efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the quality of service for all the websites using the CDN.

These rate limits will be sized according to the expected rate of requests from an average user, however the proxy system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN. When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a denial of service (DoS) attack and prevent access to the website.

Deploying mirrors for websites hosted on CDNs will require co-operation from the CDN provider. If you find that mirrors are producing many “Rate Limited Exceeded” or “Access Denied” errors then you may be suffering from this problem. Ask your administrator to configure the Bypass-Rate-Limit-Token header in the portal and at the CDN to disable the rate limiting for requests originating via the mirrors.

New Web Origin

To create a new web origin, click “Create new origin” at the top of the list. This will present you with the new origin form:

../_images/new4.png

Domain Name

Enter the domain name that you want to create a mirror for in this field. This is a required field.

Description

Enter a brief description of the website that you are creating a mirror for in this field. This is also a required field.

Group

Select the group that you want the mirror to belong to from the drop-down menu in this field. This is a required field.

Auto-Rotate

Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.

Smart Proxy

Note

This is a transitional option that will be removed once all platforms support smart proxies.

Select this field if the mirror requires a smart proxy to function properly. This is an optional field and is disabled by default. Enabling this will limit the platforms that the mirror can be deployed to as not all platforms support smart proxies at this time.

Asset Domain

Note

This is a transitional option that will be removed once all platforms support smart proxies.

Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror hosted by a smart proxy.

Static Origins

Static origins can be used to host static content at a cloud provider, and then have mirrors generated for that content rather than proxying to an external web server.

Once created a static origin will provide you with details on where and how to upload the web content.

Static origins can also be extended with additional resources:

  • Keanu Convene - A matrix-based chat platform that supports sharing of any kind. Rooms are private and encrypted by default. No software is required to join a room other than a web browser, and participants can be invited via a link.

  • Clean Insights - A private measurement platform. It is focused on assisting in answering key questions about app usage patterns, and not on enabling invasive surveillance of all user habits.

These resources can be added/removed after the creation of the static origin without having to change the static storage location or access credentials.

New Static Origin

To create a new static origin, click “Create new static origin” at the top of the list. This will present you with the new static origin form:

../_images/new5.png

Description

Enter a brief description of the static website that you are creating in this field. This is also a required field.

Group

Select the group that you want the origin to belong to from the drop-down menu in this field. This is a required field.

Auto-Rotate

Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.

Matrix Homeserver

Select the Matrix homeserver from the drop-down box to enable Keanu Convene on mirrors of this static origin.

Keanu Convene Path

Enter the subdirectory to present the Keanu Convene application at on the mirror. This defaults to “talk”.

Enable Clean Insights

When enabled, a Clean Insights Measurement Proxy endpoint is deployed on the mirror to allow for submission of results from any of the supported Clean Insights SDKs.

Web Proxies