HTTP

From Things and Stuff Wiki
Jump to navigation Jump to search


See also HTML/CSS

General

ugh. to sort.



  • https://github.com/StevenBlack/hosts - Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others. You can optionally invoke extensions to block additional sites by category.



  • RIP HTTP - HTTP is fading away, and for good reasons. But ocassionally a website that runs on HTTP can be useful, such as in captive portals. This site serves as a permanent memorial to HTTP and pledges never to switch to HTTPS. But be careful, it could have had ads or malware injected and likely was spied on, by ISPs, governments, hotspots, and other malicious actors. And it also suffers a ranking punishment.

Addressing

URI; 

<scheme name> : <hierarchical part> [ ? <query> ] [ # <fragment> ]


Headers

user-agent




  • User-Agent Client Hints - This document defines a set of Client Hints that aim to provide developers with the ability to perform agent-based content negotiation when necessary, while avoiding the historical baggage and passive fingerprinting surface exposed by the venerable `User-Agent` header.

auth

2012;

URI

  • https://en.wikipedia.org/wiki/Uniform_Resource_Identifier - a string of characters that unambiguously identifies a particular resource. To guarantee uniformity, all URIs follow a predefined set of syntax rules, but also maintain extensibility through a separately defined hierarchical naming scheme (e.g. http://).Such identification enables interaction with representations of the resource over a network, typically the World Wide Web, using specific protocols. Schemes specifying a concrete syntax and associated protocols define each URI. The most common form of URI is the Uniform Resource Locator (URL), frequently referred to informally as a web address. More rarely seen in usage is the Uniform Resource Name (URN), which was designed to complement URLs by providing a mechanism for the identification of resources in particular namespaces.


  • https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier - an internet protocol standard which builds on the Uniform Resource Identifier (URI) protocol by greatly expanding the set of permitted characters. It was defined by the Internet Engineering Task Force (IETF) in 2005 in RFC 3987. While URIs are limited to a subset of the ASCII character set, IRIs may additionally contain most characters from the Universal Character Set (Unicode/ISO 10646), including Chinese, Japanese, Korean, and Cyrillic characters.



  • https://en.wikipedia.org/wiki/Uniform_Resource_Name - a Uniform Resource Identifier (URI) that uses the urn scheme. A URI is a string of characters used to identify a name or resource. URIs are used in many Internet protocols to refer to and access information resources. URI schemes include the familiar http, as well as hundreds of others.

URNs were originally conceived to be part of a three-part information architecture for the Internet, along with Uniform Resource Locators (URLs) and Uniform Resource Characteristics (URCs), a metadata framework. As described in the 1994 RFC 1737,, and later in the 1997 RFC 2141, URNs were distinguished from URLs, which identify resources by specifying their locations in the context of a particular access protocol, such as HTTP or FTP. In contrast, URNs were conceived as persistent, location-independent identifiers assigned within defined namespaces, typically by an authority responsible for the namespace, so that they are globally unique and persistent over long periods of time, even after the resource which they identify ceases to exist or becomes unavailable.

URCs never progressed past the conceptual stage, and other technologies such as the Resource Description Framework later took their place. Since RFC 3986[5] in 2005, use of the terms "Uniform Resource Name" and "Uniform Resource Locator" has been deprecated in technical standards in favor of the term Uniform Resource Identifier (URI), which encompasses both, a view proposed in 2001 by a joint working group between the World Wide Web Consortium (W3C) and Internet Engineering Task Force (IETF).


  • https://en.wikipedia.org/wiki/Fragment_identifier - a string of characters that refers to a resource that is subordinate to another, primary resource. The primary resource is identified by a Uniform Resource Identifier (URI), and the fragment identifier points to the subordinate resource.The fragment identifier introduced by a hash mark # is the optional last part of a URL for a document. It is typically used to identify a portion of that document. The generic syntax is specified in RFC 3986. The hash-mark separator in URIs is not part of the fragment identifier.


  • https://en.wikipedia.org/wiki/Percent-encoding - also known as URL encoding, is a mechanism for encoding information in a Uniform Resource Identifier (URI) under certain circumstances. Although it is known as URL encoding, it is, in fact, used more generally within the main Uniform Resource Identifier (URI) set, which includes both Uniform Resource Locator (URL) and Uniform Resource Name (URN). As such, it is also used in the preparation of data of the application/x-www-form-urlencoded media type, as is often used in the submission of HTML form data in HTTP requests.


  • https://en.wikipedia.org/wiki/Internationalized_domain_name - an Internet domain name that contains at least one label that is displayed in software applications, in whole or in part, in a language-specific script or alphabet, such as Arabic, Chinese, Cyrillic, Devanagari, Hebrew or the Latin alphabet-based characters with diacritics or ligatures, such as French. These writing systems are encoded by computers in multibyte Unicode. Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription.


  • https://en.wikipedia.org/wiki/Punycode - a representation of Unicode with the limited ASCII character subset used for Internet hostnames. Using Punycode, host names containing Unicode characters are transcoded to a subset of ASCII consisting of letters, digits, and hyphens, which is called the Letter-Digit-Hyphen (LDH) subset. For example, München (German name for Munich) is encoded as Mnchen-3ya.While the Domain Name System (DNS) technically supports arbitrary sequences of octets in domain name labels, the DNS standards recommend the use of the LDH subset of ASCII conventionally used for host names, and require that string comparisons between DNS domain names should be case-insensitive. The Punycode syntax is a method of encoding strings containing Unicode characters, such as internationalized domain names (IDNA), into the LDH subset of ASCII favored by DNS. It is specified in IETF Request for Comments 3492.


  • http://en.wikipedia.org/wiki/Web_resource - or simply resource, is any identifiable thing, whether digital, physical, or abstract. Resources are identified using Uniform Resource Identifiers. In the Semantic Web, web resources and their semantic properties are described using the Resource Description Framework.
  • Cool URIs for the Semantic Web - The Resource Description Framework RDF allows users to describe both Web documents and concepts from the real world—people, organisations, topics, things—in a computer-processable way. Publishing such descriptions on the Web creates the Semantic Web. URIs (Uniform Resource Identifiers) are very important, providing both the core of the framework itself and the link between RDF and the Web. This document presents guidelines for their effective use. It discusses two strategies, called 303 URIs and hash URIs. It gives pointers to several Web sites that use these solutions, and briefly discusses why several other proposals have problems.


  • Sense and Reference on the Web - Harry Halpin, 2009. "This thesis builds a foundation for the philosophy of the Web by examining the crucial question: What does a Uniform Resource Identifier (URI) mean? Does it have a sense, and can it refer to things? A philosophical and historical introduction to the Web explains the primary purpose of the Web as a universal information space for naming and accessing information via URIs. A terminology, based on distinctions in philosophy, is employed to define precisely what is meant by information, language, representation, and reference. These terms are then employed to create a foundational ontology and principles of Web architecture. From this perspective, the Semantic Web is then viewed as the application of the principles of Web architecture to knowledge representation. However, the classical philosophical problems of sense and reference that have been the source of debate within the philosophy of language return. Three main positions are inspected: the logicist position, as exemplified by the descriptivist theory of reference and the first-generation Semantic Web, the direct reference position, as exemplified by Putnam and Kripke's causal theory of reference and the second-generation Linked Data initiative, and a Wittgensteinian position that views the Semantic Web as yet another public language. After identifying the public language position as the most promising, a solution of using people's everyday use of search engines as relevance feedback is proposed as a Wittgensteinian way to determine sense of URIs. This solution is then evaluated on a sample of the Semantic Web discovered by via using queries from a hypertext search engine query log. The results are evaluated and the technique of using relevance feedback from hypertext Web searches to determine relevant Semantic Web URIs in response to user queries is shown to considerably improve baseline performance. Future work for the Web that follows from our argument and experiments is detailed, and outlines of a future philosophy of the Web laid out."


CURIE

  • https://en.wikipedia.org/wiki/CURIE - or Compact URI, defines a generic, abbreviated syntax for expressing Uniform Resource Identifiers (URIs). It is an abbreviated URI expressed in a compact syntax, and may be found in both XML and non-XML grammars. A CURIE may be considered a datatype.An example of CURIE syntax: [isbn:0393315703]

URL shortners


  • tantek / Whistle - an open source, algorithmically reversible, personal URL shortener.


  • YOURLS - stands for Your Own URL Shortener. It is a small set of PHP scripts that will allow you to run your own URL shortening service (a la TinyURL or bitly).




Services


  • expiring.link - Create Temporary Links that expire after n visits or on a date

well-known

API

  • https://en.wikipedia.org/wiki/Web_API - an application programming interface for either a web server or a web browser. It is a web development concept, usually limited to a web application's client-side (including any web frameworks being used), and thus usually does not include web server or browser implementation details such as SAPIs or APIs unless publicly accessible by a remote web application.


  • https://en.wikipedia.org/wiki/Open_API - often referred to as a public API, is a publicly available application programming interface that provides developers with programmatic access to a proprietary software application or web service. APIs are sets of requirements that govern how one application can communicate and interact with another. APIs can also allow developers to access certain internal functions of a program, although this is not typically the case for web APIs. In the simplest terms, an API allows one piece of software to interact with another piece of software, whether within a single computer via a mechanism provided by the operating system or over an internal or external TCP/IP-based or non-TCP/IP-based network.[3] In the late 2010s, many APIs are provided by organisations for access with HTTP. APIs may be used by both developers inside the organisation that published the API or by any developers outside that organisation who wish to register for access to the interface.





  • Apify - Web Scraping, Data Extraction and Automation
    • https://github.com/apifytech/apify-js - The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
  • Diffbot - Knowledge Graph, AI Web Data Extraction and Crawling


RESTful

  • https://en.wikipedia.org/wiki/Representational_State_Transfer - a software architectural style that defines a set of constraints to be used for creating Web services. Web services that conform to the REST architectural style, called RESTful Web services, provide interoperability between computer systems on the Internet. RESTful Web services allow the requesting systems to access and manipulate textual representations of Web resources by using a uniform and predefined set of stateless operations. Other kinds of Web services, such as SOAP Web services, expose their own arbitrary sets of operations.

"Web resources" were first defined on the World Wide Web as documents or files identified by their URLs. However, today they have a much more generic and abstract definition that encompasses every thing or entity that can be identified, named, addressed, or handled, in any way whatsoever, on the Web. In a RESTful Web service, requests made to a resource's URI will elicit a response with a payload formatted in HTML, XML, JSON, or some other format. The response can confirm that some alteration has been made to the stored resource, and the response can provide hypertext links to other related resources or collections of resources. When HTTP is used, as is most common, the operations (HTTP methods) available are GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE. By using a stateless protocol and standard operations, RESTful systems aim for fast performance, reliability, and the ability to grow by reusing components that can be managed and updated without affecting the system as a whole, even while it is running.





  • https://github.com/Corvusoft/restbed - a comprehensive and consistent programming model for building applications that require seamless and secure communication over HTTP, with the ability to model a range of business processes, designed to target mobile, tablet, desktop and embedded production environments.



HATEOAS

  • https://en.wikipedia.org/wiki/HATEOAS - Hypermedia As The Engine Of Application State (HATEOAS) is a constraint of the REST application architecture that distinguishes it from other network application architectures. With HATEOAS, a client interacts with a network application that application servers provide dynamically entirely through hypermedia. A REST client needs no prior knowledge about how to interact with an application or server beyond a generic understanding of hypermedia. By contrast, clients and servers in some service-oriented architectures (SOA) interact through a fixed interface shared through documentation or an interface description language (IDL). The way that the HATEOAS constraint decouples client and server enables the server functionality to evolve independently.



SOAP


Google Feed API


GraphQL

  • GraphQL - a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

JSON-API


Hydra

  • Hydra Core Vocabulary - a lightweight vocabulary to create hypermedia-driven Web APIs. By specifying a number of concepts commonly used in Web APIs it enables the creation of generic API clients.

User and group

See also *nix#Users

Servers

Python 3 

python -m http.server 5674

PHP <5.4 

php -S localhost:8000

Ruby 

ruby -run -e httpd -- --port=8080 .



  • h5ai - makes browsing directories on HTTP web servers more pleasant. Directory listings get styled in a modern way and browsing through the directories is enhanced by different views, a breadcrumb and a tree overview.


Nginx

  • nginx [engine x] is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Igor Sysoev started development of Nginx in 2002, with the first public release in 2004. Nginx now hosts nearly 12.18% (22.2M) of active sites across all domains. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

Guides

Webfonts

Nginx uses a file’s mime type declaration to decide whether or not to apply compression to that file, and so we must first ensure that the four types of web font files have mime types configured.

/etc/nginx/mime.types 

application/vnd.ms-fontobject      eot;
application/x-font-ttf             ttf;
font/opentype                      ott;
font/x-woff                        woff;

remove; 

application/octet-stream          eot;
application/vnd.oasis.opendocument.text-template  ott;

Configuration

Listening on specific IP will override a wildcard IP catch-all.


Site setup

Default site folder location can vary. 

/var/www
  # debian

/etc/nginx/html
  # arch linux

Create 'Server Block' (vhost) config file in 

/etc/nginx/sites-available

and symlink to them in 

/etc/nginx/sites-enabled

Enable logging in vhost conf; 

error_log /var/log/nginx-vhostnamehere.log error;

Modules

  • Nginx modules must be selected during compile, run-time selection of modules is not currently supported.
Compression
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss
text/javascript application/json image/svg+xml application/vnd.ms-fontobject application/x-font-ttf
font/opentype;
FastCGI
Connections
  • HttpLimitConnModule - This module makes it possible to limit the number of concurrent connections for a defined key such as, for example, an ip address.
location / { 
   auth_basic            "Restricted";
   auth_basic_user_file  /etc/nginx/conf.d/htpasswd;
 }

printf "John:$(openssl passwd -crypt V3Ry)\n" >> .htpasswd # this example uses crypt encryption
printf "Mary:$(openssl passwd -apr1 SEcRe7)\n" >> .htpasswd # this example uses apr1 (Apache MD5) encryption
printf "Jane:$(openssl passwd -1 V3RySEcRe7)\n" >> .htpasswd # this example uses MD5 encryption
(PWD="SEcRe7PwD";SALT="$(openssl rand -base64 3)";SHA1=$(printf "$PWD$SALT" | openssl dgst -binary -sha1 | \ sed 's#$#'"$SALT"'#' | base64);printf "Jim:{SSHA}$SHA1\n" >> .htpasswd) # this example uses SSHA encryptio
Security
SSL
  1. Create .key and .csr
    1. openssl req -new -newkey rsa:4096 -nodes -keyout server.key -out server.csr
    2. Or using StartSSL control panel wizard, with additional key decrypt step after
  2. Provide .csr to certificate authority, certificate authority returns .crt
  3. .crt is concatenated with intermediate cert. .key can also be added; nginx will not send it
    1. cat ssl.crt sub.class1.server.ca.pem ca.pem > /etc/nginx/conf/ssl-unified.crt
  4. Nginx config points to .crt and .key

Make sure wildcard SSL is in http stanza and that any specific server is listening on 443. 

server {
   listen   80;
   listen   [::]:80;
   listen   443 default ssl;

   server_name www.example.com;

   ssl_certificate        /path/to/my/cert;
   ssl_certificate_key  /path/to/my/key;

   if ($ssl_protocol = "") {
      rewrite ^   https://$server_name$request_uri? permanent;
   }
}

ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA2:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA;
Proxy
Info
Directory listing
autoindex on;




Tools

Lua

Forks / patches

Apache

  • The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage.

.htaccess

lighttpd

Other






















  • fserv - an HTTP server that can serve static files, act as a forward and reverse HTTP proxy server, stream media content (SHOUTcast). It has a very high speed on all supported platforms due to its asynchronous I/O event architecture. It can easily process thousands of requests in parallel. The functionality of fserv can be extended by adding new modules. When configured as a proxy server, it caches responses from upstream servers on a local filesystem, dramatically speeding up the response time when the same request is received again. fserv can act as an Internet radio trasmitter or it can re-transmit other Internet radio stations. In the latter case it also can record songs into MP3 files on a local machine, which makes it a useful music-grabber.


  • ServerMe.H - An embeddable single-file C++ web server


  • webfs - a simple http server for mostly static content. You can use it to serve the content of a ftp server via http for example. It is also nice to export some files the quick way by starting a http server in a few seconds, without editing some config file first.



unix-web

CORS


  • cors - "This post aims to demystify CORS and show its lighter side–as a specification that didn’t set out to hamper the aspirations of web developers everywhere, but instead to loose us from the grip of the same-origin policy. We’ll go through each of the headers necessary to properly satisfy CORS constraints, and also discuss a couple places where CORS is now relevant but which may surprise you." [16]



Proxy

Privoxy

  • Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data and HTTP headers, controlling access, and removing ads and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. It has application for both stand-alone systems and multi-user networks.

Glype

  • Glype is a web-based proxy script written in PHP which focuses on features, functionality, and ease of use. Webmasters use Glype to quickly and easily set up their own proxy sites. Glype helps users to defeat Internet censorship and be anonymous while web browsing.

ngrok

uProxy

Lantern

dispatch-proxy

peroxide

MITM

mitmproxy

Hyperfox

Extensions





px

Testing


  • httping - like 'ping' but for http-requests. Give it an url, and it'll show you how long it takes to connect, send a request and retrieve the reply (only the headers). Be aware that the transmission across the network also takes time! So it measures the latency of the webserver + network.It supports, of course, IPv6.



  • RED - a robot that checks HTTP resources to see how they'll behave, pointing out common problems and suggesting improvements. Although it is not a HTTP conformance tester, it can find a number of HTTP-related issues.


  • Browser SOA Debugger - Depending on the view of things this is just an enhanced HTTP output formatter for tcpdump streams, or the ultimate debugger for complex HTTP oriented SOA architectures which visualizes the full HTTP interactions in a readable, reproducible way so that you can see what is actually going on in your backend.




  • Hyper - an experimental middleware architecture for HTTP servers written in PureScript. Its main focus is correctness and type-safety, using type-level information to enforce correct composition and abstraction for web servers. The Hyper project is also a breeding ground for higher-level web server constructs, which tend to fall under the “framework” category.


Clients

  • Postman - a powerful HTTP client to help test web services easily and efficiently. Postman let's you craft simple as well as complex HTTP requests quickly. It also saves requests for future use so that you never have to repeat your keystrokes ever again. Postman is designed to save you and your team tons of time. Check out more features below or just install from the Chrome Web Store to get started. [21]





Compression


Load


A/B


API

Performance

See also Server#Performance

Guides

Cookieless

Cookie free domain for static content so cooke isn't sent with request. Root domain cookies apply to all subdomain cookies, though using www. (ugh!) works. Use another domain A record to point to the site.(?)

ETag

Caching

Varnish

Load balancing

CDN

  • CoralCDN is a decentralized, self-organizing, peer-to-peer web-content distribution network. CoralCDN leverages the aggregate bandwidth of volunteers running the software to absorb and dissipate most of the traffic for web sites using the system.
.nyud.net

p2p;

Testing

  • etc.
  • GTmetrix uses Google Page Speed and Yahoo! YSlow to grade your site's performance and provides actionable recommendations to fix these issues.
  • Blitz does cloud based load and performance testing ising Sinatra, Rails and node.js.
    • Free: Sprint all you want, Rush all you want, 250 concurrent users, 1 minute rushes
  • Engulf is a scalable, distributed HTTP benchmarker, designed to let you spin up and coordinate a cluster of workers with nothing more than a single JAR. Engulf's backend is written in clojure, the frontend in javascript.

Logging

Combined Log Format 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

old skool;

Apache

Nginx

error_log  /var/log/nginx/domain.name/error.log;
access_log  /var/log/nginx/domain.name/access.log;


GoAccess

  • GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.

HAR

Control panels

Analytics

Referer

Piwik

  • Piwik is downloadable, Free/Libre (GPLv3 licensed) real time web analytics software. It provides you with detailed reports on your website visitors; the search engines and keywords they used, the language they speak, your popular pages, and much more.

WordPress

Clicky

GoatCounter

Google

Other


E-mail

Upload

Quick screipts for uploading;


WebDAV

"Linux users can mount WebDAV shares using the davfs2 and the fusedav file system modules which mount them as Coda or FUSE filesystems. KDE has native WebDAV support as part of kio_http. This enables Dolphin, Konqueror, and every other KDE application to interact directly with WebDAV servers. Nautilus also has WebDAV support built in. Many Linux distributions also include the cadaver command-line client interface, which provides an FTP-like command set. The Apache HTTP Server provides WebDAV modules based on both davfs and Apache Subversion (svn)."

File system

Davfs

fusedav

  • fusedav is a Linux userspace file system driver for mounting WebDAV shares. It makes use of FUSE as userspace file system API and neon as WebDAV API.

Software

  • neon is an HTTP and WebDAV client library, with a C interface
  • jsDAV allows you to easily add WebDAV support to a NodeJS application.


  • DAV-pocket Lab is a small project team developping WebDAV Server on Google App Engine.


  • mod_dav is an Apache module to provide DAV capabilities (RFC 2518) for your Apache web server.
  • http://www.webdav.org/goliath/
  • cadaver is a command-line WebDAV client for Unix. It supports file upload, download, on-screen display, namespace operations (move/copy), collection creation and deletion, and locking operations.


Saving

Wget

  • Wget - a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc. 
wget -O myzip.zip https://github.com/username/project/zipball/master

wget -m http://example.com
  --mirror

wget -mk http://example.com
  --convert-links

wget -mk -w 20 http://example.com
  # with delay of 20 seconds between requests

wget -E -H -k -K -p -nd http://example.com
  to mirror a single page
  --adjust-extension
  --span-hosts
  --convert-links
  --backup-converted
  --page-requisites
  --no-directories - httpd access permission issues. to try next time; -nH

wget -r -np -l 1 -Azip http://example.com/download/
  # download all links to .zip files on a given web page [27]


cURL

  • curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction. 
curl http://www.google.com/search.js -o /path/to/local/file.js

curl http://site.{one,two,three}.com

curl ftp://ftp.numericals.com/file[1-100].txt
     ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
     ftp://ftp.letters.com/file[a-z].txt
 sequences of alphanumeric series by using []

curl http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html
  Nested sequences are not supported, but you can use several ones next to each other:

curl http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt
  multiple urls + specify a step counter for the ranges to get every Nth number or letter:
  • saldl is a lightweight well-featured CLI downloader optimized for speed and early preview. based on libcurl.

Other

  • mulk - Multi-connection command line tool for downloading Internet sites with image filtering and Metalink support. Similar to wget and cURL, but it manages up to 50 simultaneous and parallel links. Main features are: HTML code parsing, recursive fetching, Metalink retrieving, segmented download and image filtering by width and height. It is based on libcurl, liburiparser, libtidy, libmetalink and libcrypto.


  • aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.


  • HTTrack allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads.
httrack "https://example.com" -O ExampleMirrorDirectory \
"-*" \
"+https://example.com/images/*" \
"-*.swf"







Scraping

See also Data#Scraping










404

Other





  • https://en.wikipedia.org/wiki/Hyper_Text_Coffee_Pot_Control_Protocol - a facetious communication protocol for controlling, monitoring, and diagnosing coffee pots. It is specified in RFC 2324, published on 1 April 1998 as an April Fools' Day RFC, as part of an April Fools prank. An extension, HTCPCP-TEA, was published as RFC 7168 on 1 April 2014 to support brewing teas, which is also an April Fools' Day RFC.

HTTP/2 (SPDY)

HTTP/3 (QUIC)



Retrieved from "https://wiki.thingsandstuff.org/index.php?title=HTTP&oldid=54552"