From fddc0d3f14f1200dc13edf5198ffa710dd8759ae Mon Sep 17 00:00:00 2001 From: Thomas Schoebel-Theuer Date: Sun, 27 Oct 2019 19:50:27 +0100 Subject: [PATCH] arch-guide: split location transparency and integrate into new locations --- docu/mars-architecture-guide.lyx | 1233 +++++++++++++++++++----------- 1 file changed, 777 insertions(+), 456 deletions(-) diff --git a/docu/mars-architecture-guide.lyx b/docu/mars-architecture-guide.lyx index 233bcce5..38f90c30 100644 --- a/docu/mars-architecture-guide.lyx +++ b/docu/mars-architecture-guide.lyx @@ -1933,6 +1933,340 @@ noprefix "false" \end_layout +\begin_layout Section +What is Location Transparency +\begin_inset CommandInset label +LatexCommand label +name "sec:Location-transparency" + +\end_inset + + +\end_layout + +\begin_layout Standard +Replication as defined in the previous section works only reasonable fast + enough when Location Transparency is implemented reasonably well, see also + section +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:Where-implement-Location-Transparency" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +. + Here is a brief explanation what it is. +\end_layout + +\begin_layout Standard +Location Transparency is an extremely important and well-known +\series bold +fundamental principle +\series default + in Distributed Systems, and has attracted research for decades. +\end_layout + +\begin_layout Standard +Simply stated, it means that the location of an object or of a service is + never (part of) a primary key, but any access is via a +\emph on +logical name +\emph default + not depending on the location. + Thus the location may (relatively easily) change at runtime. +\end_layout + +\begin_layout Standard +There are numerous examples where this fundamental principle is obeyed. + Unfortunately, there are also many examples where it is violated. +\end_layout + +\begin_layout Standard +\begin_inset Flex Custom Color Box 1 +status open + +\begin_layout Plain Layout + +\series bold +\begin_inset Argument 1 +status open + +\begin_layout Plain Layout + +\series bold +Phone numbers +\end_layout + +\end_inset + + +\series default +Phone numbers are +\emph on +not +\emph default + location transparent in general. + For stationary phones, they contain a location-dependent prefix. + In general, it is not possible to move to a different city while keeping + the old stationary phone number. + In case of mobile phones / cellphones, numbers are +\begin_inset Quotes eld +\end_inset + +more location transparent +\begin_inset Quotes erd +\end_inset + +, but even there they are +\emph on +not fully +\emph default + location transparent: for international calls, they contain prefixes referring + to the country, e.g +1 for US or +49 for Germany. + In practice, it is not easily possible to permanently move from Germany + to US, without giving up the old number after a while. + In addition, often the +\emph on +service provider +\emph default + and/or the network technology (D-net vs E-net etc) may be also be encoded + in cellphone numbers, e.g. + somewhere as an infix, so changing the provider may have some restrictions. + However, for +\emph on +most practical purposes +\emph default +, such as Europeans spending their holidays in US, mobile phone numbers + are +\emph on +sufficienctly +\emph default + location transparent. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +In practice, location transparency is not just a boolean property. + As explained by the cellphone example, it may have various +\series bold +degrees +\series default +. + In addition, it can refer to different sub-systems at different architectural + layers. + Some layers / some components may be (more or less) location transparent, + but others not at all. + Thus it is important to mention the +\series bold +layer or the component +\series default + when talking about location transparency. +\end_layout + +\begin_layout Standard +Interestingly, the Wikipedia article +\begin_inset Flex URL +status open + +\begin_layout Plain Layout + +https://en.wikipedia.org/wiki/Location_transparency +\end_layout + +\end_inset + + is an incomplete stub when this section was written (Autumn 2019). + It seems that people are actually paying less attention to it. +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Flex Custom Color Box 3 +status open + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/MatieresCorrosives.png + lyxscale 50 + scale 17 + +\end_inset + + Major violations of location transparency are almost always carrying some + +\series bold +technical debt +\series default +, likely causing future problems and impediments. +\end_layout + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +Therefore, establishment of reasonable location transparency needs to be + seen as +\series bold +best practice +\series default +. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/MatieresCorrosives.png + lyxscale 50 + scale 17 + +\end_inset + + It may happen that somebody thinks there would bot be enough time and/or + resources for implementing certain kinds of location transparency. + Although in many cases this is not really true, there might be some corner + cases where it sometimes is true, or close to true. +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Flex Custom Color Box 3 +status open + +\begin_layout Plain Layout +\noindent +I +\series bold +nvestments into location transparency +\series default + are often +\emph on +l +\series bold +ongterm +\emph default + investments +\series default +. + Not doing it will likely +\series bold +decrease your business opportunities +\series default + and +\series bold +increase your risks +\series default + in the long term. +\end_layout + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +Location transparency is simply a certain type of +\series bold +redirection mechanism +\series default +, which +\series bold +\emph on +automatically +\emph default + follows the current location +\series default + of a service and/or its storage. + It makes you +\emph on +independent +\emph default + from various placement strategies. +\end_layout + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +Once you have established location transparency, a +\series bold +multitude of placement strategies +\series default + for your services and/or your storage locations is possible. + This opens up more +\series bold +opportunities +\series default + for +\series bold +higher efficiency +\series default +. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +If anyone is arguing that location transparancy were +\emph on +not needed +\emph default + as a major feature, you should check whether such a person is really an + expert. + There needs to be a clear and valid justification for such an opinion. +\end_layout + +\begin_layout Standard +Hints for implementation of location transparency are in section +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:Where-implement-Location-Transparency" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +. +\end_layout + \begin_layout Section What is HA = High Availability \begin_inset CommandInset label @@ -4456,453 +4790,6 @@ any type of clusters and/or grids. \end_layout -\begin_layout Section -Location Transparency -\begin_inset CommandInset label -LatexCommand label -name "sec:Location-transparency" - -\end_inset - - -\end_layout - -\begin_layout Standard -Location Transparency is an extremely important and well-known -\series bold -fundamental principle -\series default - in Distributed Systems, and has attracted research for decades. -\end_layout - -\begin_layout Standard -Simply stated, it means that the location of an object or of a service is - never (part of) a primary key, but any access is via a -\emph on -logical name -\emph default - not depending on the location. - Thus the location may (relatively easily) change at runtime. -\end_layout - -\begin_layout Standard -There are numerous examples where this fundamental principle is obeyed. - Unfortunately, there are also many examples where it is violated. -\end_layout - -\begin_layout Standard -\begin_inset Flex Custom Color Box 1 -status open - -\begin_layout Plain Layout - -\series bold -\begin_inset Argument 1 -status open - -\begin_layout Plain Layout - -\series bold -Phone numbers -\end_layout - -\end_inset - - -\series default -Phone numbers are -\emph on -not -\emph default - location transparent in general. - For stationary phones, they contain a location-dependent prefix. - In general, it is not possible to move to a different city while keeping - the old stationary phone number. - In case of mobile phones / cellphones, numbers are -\begin_inset Quotes eld -\end_inset - -more location transparent -\begin_inset Quotes erd -\end_inset - -, but even there they are -\emph on -not fully -\emph default - location transparent: for international calls, they contain prefixes referring - to the country, e.g +1 for US or +49 for Germany. - In practice, it is not easily possible to permanently move from Germany - to US, without giving up the old number after a while. - In addition, often the -\emph on -service provider -\emph default - and/or the network technology (D-net vs E-net etc) may be also be encoded - in cellphone numbers, e.g. - somewhere as an infix, so changing the provider may have some restrictions. - However, for -\emph on -most practical purposes -\emph default -, such as Europeans spending their holidays in US, mobile phone numbers - are -\emph on -sufficienctly -\emph default - location transparent. -\end_layout - -\end_inset - - -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Graphics - filename images/lightbulb_brightlit_benj_.png - lyxscale 12 - scale 7 - -\end_inset - -In practice, location transparency is not just a boolean property. - As explained by the cellphone example, it may have various -\series bold -degrees -\series default -. - In addition, it can refer to different sub-systems at different architectural - layers. - Some layers / some components may be (more or less) location transparent, - but others not at all. - Thus it is important to mention the -\series bold -layer or the component -\series default - when talking about location transparency. -\end_layout - -\begin_layout Standard -Interestingly, the Wikipedia article -\begin_inset Flex URL -status open - -\begin_layout Plain Layout - -https://en.wikipedia.org/wiki/Location_transparency -\end_layout - -\end_inset - - is an incomplete stub when this section was written (Autumn 2019). - It seems that people are actually paying less attention to it. -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Flex Custom Color Box 3 -status open - -\begin_layout Plain Layout -\noindent -\begin_inset Graphics - filename images/MatieresCorrosives.png - lyxscale 50 - scale 17 - -\end_inset - - Major violations of location transparency are almost always carrying some - -\series bold -technical debt -\series default -, likely causing future problems and impediments. -\end_layout - -\begin_layout Plain Layout -\noindent -\begin_inset Graphics - filename images/lightbulb_brightlit_benj_.png - lyxscale 12 - scale 7 - -\end_inset - -Therefore, establishment of location transparency needs to be seen as -\series bold -best practice -\series default -. -\end_layout - -\end_inset - - -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Graphics - filename images/MatieresCorrosives.png - lyxscale 50 - scale 17 - -\end_inset - - It may happen that somebody thinks there would be not enough time and/or - resources for implementing certain kinds of location transparency. - Although in many cases this is not really true, there might be some corner - cases where it sometimes is true, or close to true. - Notice that -\series bold -investments into location transparency -\series default - are often -\emph on -longterm -\emph default - investments. - Not doing it will likely -\series bold -decrease your business opportunities -\series default - and -\series bold -increase your risks -\series default - in the long term. - If anyone is arguing that location transparancy were -\emph on -not needed -\emph default - as a major feature, you should check whether such a person is really an - expert. - There needs to be a clear and valid justification for such an opinion. -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Flex Custom Color Box 2 -status open - -\begin_layout Plain Layout -\noindent -\begin_inset Argument 1 -status open - -\begin_layout Plain Layout - -\series bold -Where location transparency makes sense or not -\end_layout - -\end_inset - - -\begin_inset Graphics - filename images/lightbulb_brightlit_benj_.png - lyxscale 12 - scale 7 - -\end_inset - -In general, it is not necessary to implement location transparency -\emph on -everywhere -\emph default -, for each and every single component / subsystem. - The art of system architecture consists of knowing -\end_layout - -\begin_layout Enumerate -\noindent -where it is -\emph on -needed -\emph default -, -\end_layout - -\begin_layout Enumerate -\noindent -where it is -\emph on -beneficial -\emph default - for future growth / future reqirements in multiple dimensions, -\end_layout - -\begin_layout Enumerate -\noindent -where it is (or will be) too expensive to pay off in the mid-term future, - using current technology, but nevertheless -\emph on -cheap provisions for its later introduction -\emph default - can be prepared, and -\end_layout - -\begin_layout Enumerate -\noindent -where its lack can be easily (or even -\emph on -trivially -\emph default -) compensated by location transparency at another layer, such that a particular - component does not need to be constructed with location transparency, but - nevertheless the -\emph on -overall system -\emph default - is sufficiently location transparent, and -\end_layout - -\begin_layout Enumerate -when there are multiple choices -\emph on -where -\emph default - to implement it, knowing which will be the best one for a familiy of use - cases, and finally -\end_layout - -\begin_layout Enumerate - -\emph on -how -\emph default - to implement it. - For example, a common misconception is to believe that storage must always - reside at a storage network. - Football (see -\family typewriter -football-user-manual.pdf -\family default -) demonstrates that sufficient -\begin_inset Foot -status open - -\begin_layout Plain Layout -There could be arguments that Football's background migrations might be - too slow or might take too long for certain use cases. - Notice that -\family typewriter -BigCluster -\family default - also needs data migration during operations, e.g. - upon replacement of physical disks. - When the -\family typewriter -FlexibleSharding -\family default - model (see section -\begin_inset CommandInset ref -LatexCommand nameref -reference "subsec:FlexibleSharding" -plural "false" -caps "false" -noprefix "false" - -\end_inset - -) is combined with Football, it provides practically the same timescale - and flexibility than -\family typewriter -BigCluster -\family default -. -\end_layout - -\end_inset - - location transparency can be achieved on top of local storage, while expensive - and performance-eating dedicated storage networks -\begin_inset Foot -status open - -\begin_layout Plain Layout -Anyway, realtime storage networks cannot span long distances. - Thus they are not suitable for achieving location transparency in a geo-redunda -nt setup. -\end_layout - -\end_inset - - are not generally necessary for achieving location transparency. -\end_layout - -\end_inset - - -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Graphics - filename images/lightbulb_brightlit_benj_.png - lyxscale 12 - scale 7 - -\end_inset - -In the definition of Cloud Storage in section -\begin_inset CommandInset ref -LatexCommand vref -reference "sec:Requirements-for-Cloud" -plural "false" -caps "false" -noprefix "false" - -\end_inset - -, the requirement -\begin_inset Quotes eld -\end_inset - -act as one -\begin_inset Quotes erd -\end_inset - - is -\emph on -implying -\emph default - some appropriate type of location transparency of the resources. -\end_layout - -\begin_layout Standard -\noindent -\begin_inset Graphics - filename images/MatieresCorrosives.png - lyxscale 50 - scale 17 - -\end_inset - - Consequence: any system not sufficiently implementing location transparency - of the customer's resources (visible layer from outside) should not be - called -\begin_inset Quotes eld -\end_inset - -Cloud Storage -\begin_inset Quotes erd -\end_inset - - or a -\begin_inset Quotes eld -\end_inset - -Cloud Product -\begin_inset Quotes erd -\end_inset - - when location transparency is not sufficient at the layer of the customer. -\end_layout - \begin_layout Section Layering Rules and their Importance \begin_inset CommandInset label @@ -7837,6 +7724,328 @@ name "subsec:Flexibility-of-Failover" \end_layout +\begin_layout Standard +This section is also relevant for +\series bold +networking departments +\series default + and their +\series bold +management +\series default + in a bigger enterprise. +\end_layout + +\begin_layout Standard +There are two important properties of replication handover / failover: +\end_layout + +\begin_layout Enumerate + +\series bold +Timely behaviour +\series default +: how fast can it be done? +\end_layout + +\begin_layout Enumerate +What is the +\series bold +granularity +\series default +: which are the items that can be switched? +\end_layout + +\begin_layout Enumerate +Physical +\series bold +distance +\series default +: both geo-redundancy (see section +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:What-is-Geo-Redundancy" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +) and cross-datacenter replication, even when the latter is only over short + distances, are requiring different +\series bold +network support +\series default + than simple handover / failover in the same rack. +\end_layout + +\begin_layout Standard +All of these aspects are only reasonable to implement via Location Transparency. + Location Transparency has been introduced in section +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:Location-transparency" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +. + Here we look into some details how to implement it. +\end_layout + +\begin_layout Subsection +Where to implement Location Transparency +\begin_inset CommandInset label +LatexCommand label +name "sec:Where-implement-Location-Transparency" + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Flex Custom Color Box 2 +status open + +\begin_layout Plain Layout +\noindent +\begin_inset Argument 1 +status open + +\begin_layout Plain Layout + +\series bold +Where location transparency makes sense or not +\end_layout + +\end_inset + + +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +In general, it is not necessary to implement location transparency +\emph on +everywhere +\emph default +, for each and every single component / subsystem. + The art of system architecture consists of knowing +\end_layout + +\begin_layout Enumerate +\noindent +where it is +\emph on +needed +\emph default +, +\end_layout + +\begin_layout Enumerate +\noindent +where it is +\emph on +beneficial +\emph default + for future growth / future reqirements in multiple dimensions, +\end_layout + +\begin_layout Enumerate +\noindent +where it is (or will be) too expensive to pay off in the mid-term future, + using current technology, but nevertheless +\emph on +cheap provisions for its later introduction +\emph default + can be prepared, and +\end_layout + +\begin_layout Enumerate +\noindent +where its lack can be easily (or even +\emph on +trivially +\emph default +) compensated by location transparency at another layer, such that a particular + component does not need to be constructed with location transparency, but + nevertheless the +\emph on +overall system +\emph default + is sufficiently location transparent, and +\end_layout + +\begin_layout Enumerate +when there are multiple choices +\emph on +where +\emph default + to implement it, knowing which will be the best one for a familiy of use + cases, and finally +\end_layout + +\begin_layout Enumerate + +\emph on +how +\emph default + to implement it. + For example, a common misconception is to believe that storage must always + reside at a storage network. + Football (see +\family typewriter +football-user-manual.pdf +\family default +) demonstrates that sufficient +\begin_inset Foot +status open + +\begin_layout Plain Layout +There could be arguments that Football's background migrations might be + too slow or might take too long for certain use cases. + Notice that +\family typewriter +BigCluster +\family default + also needs data migration during operations, e.g. + upon replacement of physical disks. + When the +\family typewriter +FlexibleSharding +\family default + model (see section +\begin_inset CommandInset ref +LatexCommand nameref +reference "subsec:FlexibleSharding" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +) is combined with Football, it provides practically the same timescale + and flexibility than +\family typewriter +BigCluster +\family default +. +\end_layout + +\end_inset + + location transparency can be achieved on top of local storage, while expensive + and performance-eating dedicated storage networks +\begin_inset Foot +status open + +\begin_layout Plain Layout +Anyway, realtime storage networks cannot span long distances. + Thus they are not suitable for achieving location transparency in a geo-redunda +nt setup. +\end_layout + +\end_inset + + are not generally necessary for achieving location transparency. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +In the definition of Cloud Storage in section +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:Requirements-for-Cloud" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +, the requirement +\begin_inset Quotes eld +\end_inset + +act as one +\begin_inset Quotes erd +\end_inset + + is +\emph on +implying +\emph default + some appropriate type of location transparency of the resources. +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Flex Custom Color Box 3 +status open + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/MatieresCorrosives.png + lyxscale 50 + scale 17 + +\end_inset + + Consequence: any system not sufficiently implementing location transparency + of the customer's resources (visible layer from outside) should not be + called +\begin_inset Quotes eld +\end_inset + +Cloud Storage +\begin_inset Quotes erd +\end_inset + + or a +\begin_inset Quotes eld +\end_inset + +Cloud Product +\begin_inset Quotes erd +\end_inset + + when location transparency is not sufficient from the viewpoint of customers. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +In the rest of this section, we concetrate on cross-datacenter replication + scenarios, including geo-redundancy. +\end_layout + +\begin_layout Subsection +Granularity of Cross-Datacenter and Geo-Redundant Handover / Failover +\end_layout + \begin_layout Standard Typical management buzzwords like DR = Disaster Recovery or CDP = Continuous Data Protection are neglecting the @@ -7868,6 +8077,27 @@ Replication means that both the data and the necessary compute resources are available in two datacenters. + See also sections +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:What-is-Backup" +plural "false" +caps "false" +noprefix "false" + +\end_inset + + and +\begin_inset CommandInset ref +LatexCommand nameref +reference "sec:Replication-vs-Backup" +plural "false" +caps "false" +noprefix "false" + +\end_inset + +. \end_layout \end_inset @@ -8393,8 +8623,8 @@ low MTTR \begin_layout Standard \noindent -In order to implement flexibility of handover / failover, the network infrastruc -ture (as well as other infrastructures) must support it. +In order to implement good flexibility of handover / failover, the network + infrastructure (as well as other infrastructures) must support it. Here are \series bold levels of flexibility @@ -8468,12 +8698,16 @@ ignored \end_inset Dynamic routing protocols at AS = Autonomous Systems level are your friend, - such as BGP = Border Gateway Protocol. + such as +\series bold +BGP = Border Gateway Protocol +\series default +. For any \emph on serious \emph default - geo-redundancy, it is a + cross-datacenter scenarios and/or geo-redundancy, it is a \series bold must \series default @@ -8541,6 +8775,57 @@ simple of building blocks, or even per hypervisor / physical server. When automated appropriately, switchover is a matter of minutes, or even seconds. +\begin_inset Newline newline +\end_inset + + +\begin_inset Flex Custom Color Box 3 +status open + +\begin_layout Plain Layout +\begin_inset Argument 1 +status open + +\begin_layout Plain Layout + +\series bold +Requirements for networking +\end_layout + +\end_inset + +Starting with this level of flexibility, +\series bold +BGP +\series default + = Border Gateway Protocol or similar network protocols are a +\series bold +must +\series default +. +\end_layout + +\begin_layout Plain Layout +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + + Anyway, when you have the effort of BGP implementation for this level, + consider to +\series bold +do it right from scratch +\series default +. + Also support the following better levels from the network side of the company. +\end_layout + +\end_inset + + \end_layout \begin_layout Enumerate @@ -8619,8 +8904,11 @@ There is no 100% DDOS protection. \end_layout \begin_layout Enumerate -(most flexible) In addition to dynamic routing at VM level, the VMs are - +(most flexible) In addition to dynamic routing at VM level, the VMs +\emph on +themselves +\emph default + are \series bold location transparent \series default @@ -8648,6 +8936,18 @@ status open \begin_layout Plain Layout \noindent +\begin_inset Argument 1 +status open + +\begin_layout Plain Layout + +\series bold +Recommended flexibility +\end_layout + +\end_inset + + \begin_inset Graphics filename images/lightbulb_brightlit_benj_.png lyxscale 12 @@ -8655,7 +8955,7 @@ status open \end_inset -The + The \series bold ability for butterfly \series default @@ -8704,8 +9004,24 @@ y. \end_layout \begin_layout Standard -Example: 1&1 Ionos ShaHoLin = Shared Hosting Linux has implemented it on - thousands of servers, and on several petabytes of data. +\begin_inset Flex Custom Color Box 1 +status open + +\begin_layout Plain Layout +\begin_inset Argument 1 +status open + +\begin_layout Plain Layout + +\series bold +Ability for butterfly +\end_layout + +\end_inset + +1&1 Ionos ShaHoLin = Shared Hosting Linux has implemented the ability for + butterfly via BGP location transparency on thousands of servers, and on + several petabytes of data. See \begin_inset CommandInset ref LatexCommand nameref @@ -8719,6 +9035,11 @@ noprefix "false" . \end_layout +\end_inset + + +\end_layout + \begin_layout Section Point-in-time Replication via ZFS Snapshots \begin_inset CommandInset label