Web Mirrors =========== Background ---------- A web mirror can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable. Web proxies work by forwarding requests to the original website, and providing a different URL to access that content. By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website's URL. Web mirrors also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content. This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the deployment of new URLs, and for the :doc:`distribution lists ` to allow applications and end-users to discover new URLs. Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge. Architecture ------------ Before configuring any web origins to create mirrors, you must first configure an origin group. The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration. The shared configuration will include which pools will require mirrors for the origins. For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed. This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors. Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality. Web origins represent the source website that has been blocked. Web Origins ----------- Rate Limiting ^^^^^^^^^^^^^ CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure that the network resources are efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the quality of service for all the websites using the CDN. These rate limits will be sized according to the expected rate of requests from an average user, however the mirror system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN. When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a denial of service (DoS) attack and prevent access to the website. Deploying mirrors for websites hosted on CDNs will require co-operation from the CDN provider. If you find that mirrors are producing many "Rate Limited Exceeded" or "Access Denied" errors then you may be suffering from this problem. Ask your administrator to configure the `Bypass-Rate-Limit-Token` header in the portal and at the CDN to disable the rate limiting for requests originating via the mirrors. New Web Origin ^^^^^^^^^^^^^^ To create a new web origin, click "Create new origin" at the top of the list. This will present you with the new origin form: .. image:: /_static/origin/new.png :width: 800 Domain Name """"""""""" Enter the domain name that you want to create a mirror for in this field. This is a required field. Description """"""""""" Enter a brief description of the website that you are creating a mirror for in this field. This is also a required field. Group """"" Select the group that you want the mirror to belong to from the drop-down menu in this field. This is a required field. Auto-Rotate """"""""""" Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default. Asset Domain """""""""""" Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror hosted by a smart proxy.