HTTP

From Things and Stuff Wiki
Revision as of 20:09, 16 March 2018 by Milk (talk | contribs) (→‎Other)
Jump to navigation Jump to search


See also HTML/CSS

to sort

Addressing

URI; 

<scheme name> : <hierarchical part> [ ? <query> ] [ # <fragment> ]


Headers

user-agent

auth

2012;

URLs

  • YOURLS stands for Your Own URL Shortener. It is a small set of PHP scripts that will allow you to run your own URL shortening service (a la TinyURL or bitly).




Services

User and group

See also *nix#Users

Servers

Python 2 

 python -m SimpleHTTPServer 8000

Python 3 

python -m http.server 5674

PHP <5.4 

php -S localhost:8000

Ruby 

ruby -run -e httpd -- --port=8080 .


  • h5ai makes browsing directories on HTTP web servers more pleasant. Directory listings get styled in a modern way and browsing through the directories is enhanced by different views, a breadcrumb and a tree overview.

Nginx

  • nginx [engine x] is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Igor Sysoev started development of Nginx in 2002, with the first public release in 2004. Nginx now hosts nearly 12.18% (22.2M) of active sites across all domains. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

Guides

Webfonts

Nginx uses a file’s mime type declaration to decide whether or not to apply compression to that file, and so we must first ensure that the four types of web font files have mime types configured.

/etc/nginx/mime.types 

application/vnd.ms-fontobject      eot;
application/x-font-ttf             ttf;
font/opentype                      ott;
font/x-woff                        woff;

remove; 

application/octet-stream          eot;
application/vnd.oasis.opendocument.text-template  ott;

Configuration

Listening on specific IP will override a wildcard IP catch-all.

Site setup

Default site folder location can vary. 

/var/www
  # debian

/etc/nginx/html
  # arch linux

Create 'Server Block' (vhost) config file in 

/etc/nginx/sites-available

and symlink to them in 

/etc/nginx/sites-enabled

Enable logging in vhost conf; 

error_log /var/log/nginx-vhostnamehere.log error;

Modules

  • Nginx modules must be selected during compile, run-time selection of modules is not currently supported.
Compression
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss
text/javascript application/json image/svg+xml application/vnd.ms-fontobject application/x-font-ttf
font/opentype;
FastCGI
Connections
  • HttpLimitConnModule - This module makes it possible to limit the number of concurrent connections for a defined key such as, for example, an ip address.
location / { 
   auth_basic            "Restricted";
   auth_basic_user_file  /etc/nginx/conf.d/htpasswd;
 }

printf "John:$(openssl passwd -crypt V3Ry)\n" >> .htpasswd # this example uses crypt encryption
printf "Mary:$(openssl passwd -apr1 SEcRe7)\n" >> .htpasswd # this example uses apr1 (Apache MD5) encryption
printf "Jane:$(openssl passwd -1 V3RySEcRe7)\n" >> .htpasswd # this example uses MD5 encryption
(PWD="SEcRe7PwD";SALT="$(openssl rand -base64 3)";SHA1=$(printf "$PWD$SALT" | openssl dgst -binary -sha1 | \ sed 's#$#'"$SALT"'#' | base64);printf "Jim:{SSHA}$SHA1\n" >> .htpasswd) # this example uses SSHA encryptio
Security
SSL
  1. Create .key and .csr
    1. openssl req -new -newkey rsa:4096 -nodes -keyout server.key -out server.csr
    2. Or using StartSSL control panel wizard, with additional key decrypt step after
  2. Provide .csr to certificate authority, certificate authority returns .crt
  3. .crt is concatenated with intermediate cert. .key can also be added; nginx will not send it
    1. cat ssl.crt sub.class1.server.ca.pem ca.pem > /etc/nginx/conf/ssl-unified.crt
  4. Nginx config points to .crt and .key

Make sure wildcard SSL is in http stanza and that any specific server is listening on 443. 

server {
   listen   80;
   listen   [::]:80;
   listen   443 default ssl;

   server_name www.example.com;

   ssl_certificate        /path/to/my/cert;
   ssl_certificate_key  /path/to/my/key;

   if ($ssl_protocol = "") {
      rewrite ^   https://$server_name$request_uri? permanent;
   }
}

ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA2:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA;
Proxy
Info
Directory listing
autoindex on;

Tools

Lua

Forks / patches

Apache

  • The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage.

.htaccess

lighttpd

Other




  • fserv - an HTTP server that can serve static files, act as a forward and reverse HTTP proxy server, stream media content (SHOUTcast). It has a very high speed on all supported platforms due to its asynchronous I/O event architecture. It can easily process thousands of requests in parallel. The functionality of fserv can be extended by adding new modules. When configured as a proxy server, it caches responses from upstream servers on a local filesystem, dramatically speeding up the response time when the same request is received again. fserv can act as an Internet radio trasmitter or it can re-transmit other Internet radio stations. In the latter case it also can record songs into MP3 files on a local machine, which makes it a useful music-grabber.

unix-web

CORS

Proxy

  • Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data and HTTP headers, controlling access, and removing ads and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. It has application for both stand-alone systems and multi-user networks.
  • HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net
  • Glype is a web-based proxy script written in PHP which focuses on features, functionality, and ease of use. Webmasters use Glype to quickly and easily set up their own proxy sites. Glype helps users to defeat Internet censorship and be anonymous while web browsing.

Extensions

Testing

  • RED is a robot that checks HTTP resources to see how they'll behave, pointing out common problems and suggesting improvements. Although it is not a HTTP conformance tester, it can find a number of HTTP-related issues.
  • Browser SOA Debugger - Depending on the view of things this is just an enhanced HTTP output formatter for tcpdump streams, or the ultimate debugger for complex HTTP oriented SOA architectures which visualizes the full HTTP interactions in a readable, reproducible way so that you can see what is actually going on in your backend.
  • Postman is a powerful HTTP client to help test web services easily and efficiently. Postman let's you craft simple as well as complex HTTP requests quickly. It also saves requests for future use so that you never have to repeat your keystrokes ever again. Postman is designed to save you and your team tons of time. Check out more features below or just install from the Chrome Web Store to get started. [17]


Compression

Load

A/B

Performance

See also Server#Performance

Guides

Cookieless

Cookie free domain for static content so cooke isn't sent with request. Root domain cookies apply to all subdomain cookies, though using www. (ugh!) works. Use another domain A record to point to the site.(?)

ETag

Caching

Varnish

Load balancing

CDN

  • CoralCDN is a decentralized, self-organizing, peer-to-peer web-content distribution network. CoralCDN leverages the aggregate bandwidth of volunteers running the software to absorb and dissipate most of the traffic for web sites using the system.
.nyud.net

p2p;

Testing

  • etc.
  • GTmetrix uses Google Page Speed and Yahoo! YSlow to grade your site's performance and provides actionable recommendations to fix these issues.
  • Blitz does cloud based load and performance testing ising Sinatra, Rails and node.js.
    • Free: Sprint all you want, Rush all you want, 250 concurrent users, 1 minute rushes
  • Engulf is a scalable, distributed HTTP benchmarker, designed to let you spin up and coordinate a cluster of workers with nothing more than a single JAR. Engulf's backend is written in clojure, the frontend in javascript.

Logging

Combined Log Format 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

old skool;

Apache

Nginx

error_log  /var/log/nginx/domain.name/error.log;
access_log  /var/log/nginx/domain.name/access.log;


GoAccess

  • GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.

HAR

Control panels

Analytics

Referer

Piwik

  • Piwik is downloadable, Free/Libre (GPLv3 licensed) real time web analytics software. It provides you with detailed reports on your website visitors; the search engines and keywords they used, the language they speak, your popular pages, and much more.

WordPress

Clicky

Google

Other


E-mail

Upload

Quick screipts for uploading;

Saving

wget

wget -r -np -l 1 -A zip http://example.com/download/
  # download all links to .zip files on a given web page [22]

wget -O myzip.zip https://github.com/zeromq/jzmq/zipball/master

wget -m http://example.com
  --mirror

wget -mk http://example.com
  --convert-links

wget -mk -w 20 http://example.com
  with delay between requests



wget -E -H -k -K -p -nd http://example.com
  to mirror a single page
  --adjust-extension
  --span-hosts
  --convert-links
  --backup-converted
  --page-requisites
  --no-directories - httpd access permission issues. to try next time; -nH


cURL

  • curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction. 
curl http://www.google.com/search.js -o /path/to/local/file.js

curl http://site.{one,two,three}.com

curl ftp://ftp.numericals.com/file[1-100].txt
     ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
     ftp://ftp.letters.com/file[a-z].txt
 sequences of alphanumeric series by using []

curl http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html
  Nested sequences are not supported, but you can use several ones next to each other:

curl http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt
  multiple urls + specify a step counter for the ranges to get every Nth number or letter:
  • saldl is a lightweight well-featured CLI downloader optimized for speed and early preview. based on libcurl.

other

  • mulk - Multi-connection command line tool for downloading Internet sites with image filtering and Metalink support. Similar to wget and cURL, but it manages up to 50 simultaneous and parallel links. Main features are: HTML code parsing, recursive fetching, Metalink retrieving, segmented download and image filtering by width and height. It is based on libcurl, liburiparser, libtidy, libmetalink and libcrypto.
  • aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.
  • HTTrack allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads.
httrack "https://example.com" -O ExampleMirrorDirectory \
"-*" \
"+https://example.com/images/*" \
"-*.swf"

Scraping

See also Data#Scraping



404

Other

HTTP/2

see SPDY and such conversation

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=HTTP&oldid=45131"