Optimistic Mechanisms

• Optimistic mechanism allow all LPs proceed because it is possible that causality error might not occur.

• If at a later stage an causality error is detected, the recover process takes place, undoing the effects of the events that have been processed (rollback).

• The event causing rollback is called a straggler.

• An event may do two things that have to be rolled back:
  1. it may modify the state of the logical process;
  2. it may send event messages to other processes

• Rolling back the state is accomplished by periodically saving the process's state, and restoring an old state vector on rollback. The state has to be rolled back to a simulation time that is equal or smaller than the time of the straggler.

• "Unsending" a previously sent message is accomplished by sending a negative message or an anti-message that annihilates the original message when it reaches its destination.

• If a process receives an anti-message that corresponds to a positive message that has already been processed, this anti-message becomes a straggler, causing rollback on this LP.

• Recursively repeating this procedure allows all the effect of erroneous computation to eventually be canceled. It can be proved that this process converges, and it always makes progress under certain conditions.

• The fundation of the optimistic approach is virtual time, which is one of the most important concepts in distributed computing.
  • A virtual time system is a distributed system executing in coordination with an imaginary virtual clock that ticks virtual time.

  • Virtual time itself is a global, one-dimentional, temporary coordinate system imposed on a distributed computation.
    • It is used to measure computational progress and to define synchronization.
    • It may or may not have a connection with real time.
    • It is a real positive value, totally ordered by $<$.

  • Virtual time systems are subject to two fundamental rules.
    1. The virtual send time of each message must be less than its virtual receive time.
    2. The virtual time of each event in a process must be less than the virtual time of the next event at that process.
    These rules are exactly what Lamport's Clock Conditions (the well-known Happens-Before relation).

  • The major constraint on the implementation of virtual time can be stated: If an event A causes event B, then the exeuction of A and B must be scheduled in real time so that A is completed before B starts.

    This implies that if A does not cause B, B could be executed before A in real time, even if the logical time of B is after that of A.

  • One implementation of virtual time is the Time Warp system.
    • For correct implementation of virtual time, it is necessary and sufficient that at each process messages are handled in timestamp order.
    • Structure of the run time representation.
      • A unique process name.
      • A local virtual clock (LVT) which has to be compatiable with the global virtual time (GVT), but it doesn't have to have the same value as GVT.
      • A state which is a collection of variables.
      • A state queue containing saved copies of the process's recent states.
      • An input queue containting all recently arrived messages sorted in order of virtual receive time.
      • An output queue containing all negative copies of the messages this process has recently sent, kept in virtual send time order. They will be used in case of rollback.
    • See Figure 1 and Figure 2 in Jefferson's 1985 paper for illustration.
    • Global virtual time: GVT at real time r is the minimum of (1) all local virtual times in all local virtual clock at time r, and (2) of the virtual send times of all messages that are in transition.
      • GVT never decreases.
      • GVT can serve as a floor for the virtual times to which any process can ever again roll back.
      • GVT can be viewed as a moving commitment horizon: any event with virtual time less than GVT cannot be rolled back and may be removed from the system.

        Thus the name time warp.

      • If (1) every event completes normally, and (2) messages are delivered reliably, and (3) the scheduler does not indefinitely postpone execution of the farthest-behind process, and (4) there is sufficient memory, then GVT must eventually increase.

  • Examples of virtual time system include distributed discrete event simulation, distrbuted database concurrency control, virtual circuit communicaiton.

  • Virtual time has an anlogy to virtual memory in memory management.
    • The virtual address space of a page is its spatial coordinate; the virtual time of an event is its temporal coordinate.
    • A page resident in main memory at time t is analogous to an event with a virtual time in the future of process x, where the page may be accessed in the future, the event will be processed in the future.
    • A page out of memory at time t is analogous to an event in the present or past of process x.
    • Accessing a page in memory is relatively inexpensive, but accessing a page out of memory at time t is very expensive (page fault); similarily sending a message that arrives in the virtual future of the receiving process is relatively inexpensive, while sending a message into its virtual past causes a very expensive time fault, that is, rollback.
    • Under a virtual memory system, it is only cost-effective in time to execute programs that obey the spatial locality principle, so that most memory accesses are to pages already resident in memory, and page faults are relatively rare. Likewise, under a virtual time system, it is only cost-effective to run programs that obey temporal locality principle, that is, most messages arrive in the virtual future of the destination processes so that time faults are relatively rare.
    • The term ``memory mapping'' refers to the translation of virtual addresses to real address. We could use time mapping to refer to the mapping of virtual times to real times. The same virtual address may be mapped to different physical address in memory at different time, and similarily, the same virtual time may be mapped to (scheduled at) different real times at different places.
    • The only acceptable memory maps are the one-to-one functions because they preserve distinctness, mapping distinct virtual addresses (pages) to distinct real addresses (frames). At any given moment, some virtual addresses may be unmapped because they refer to pages not in memory. Similarily the only acceptable time maps are the strictly increasing function because they preserve order, mapping distinct virtual times into disctinct real times. At any given place, some events may be unmapped (not yet scheduled) because they are in the local future.
    • For a process running under virtual memory, the page fault rate can usually be reduced by increasing the number of pages it is permitted to have in main memory. Similarily the time fault rate of a process running under a virtual time can usually be reduced by increasing the number of events that are in its near future. Essentially, more interaction among LPs reduces the chances of time fault. But it may slow down the simulation progress.
    • If a process can have enough pages in memory, its page fault rate can be reduced to zero; in general, this is undesirable because it leads to inefficient use of main memory. Similarily, if a process has sufficiently large number of events to process in a short period of time, its time fault rate can be reduced to zero; this too is undesirable because this process then becomes bottleneck, holding back the progress of GVT, leading to inefficient use of real time (other processes have to wait).

• GVT calculation. There are many different ways to compute GVT.
  • Centralized calculation: Each LP sends its LVT to a centralized manager, the minimum of these is the calculated GVT.

  • Distributed calculation: LPs broadcast their LVT periodically. Each LP calculates GVT to be the minimum of all LVTs (need non-blocking read).

• Rollback distance vs. forwarding distance (a model of random walk)
  • Rollback converges (partial ordering of the events, cases for events that have the same time stamp, and different time stamp)

• Memory management, all issues are memory vs. speed
  • Interleaved state saving (How often?) Instead of saving every single input/output events, interleaved state saving strategy only saves a snap shot of the state at certain time. When a rollback is needed, the simulation is rolled back to the point where the state is saved.
  • Fossil collection: events that have a clock value smaller than GVT will never be used again, removing them can recover some memory
    • lazy: don't collect until running our memory
    • aggressive: collect at a fixed interval or upon a certain event

  • Return un-processed messages

    When memory is running low, one can return un-processed input events back to the sender to free up some memory. The ones with highest time stamps should be returned first because they are less likely to have effects on the system.

  • Gafni's protocol: instead of always returning un-processed message, reclaiming memory according to the nature of the activity.
    • If an output event is the cause, a corresponding anti-event is sent. The local simulation is restored to a previous state.
    • If an input event is the cause, do conventional return-message.
    • If a state saving is the cause, discard that set of state variables. They can be re-computed later if needed.
    • Artificial rollback: with the absence of straggler, rollback is called artificially to reclaim memory.

• Communications in PDES
  • Communication overhead
  • Difference between parallel simulation (shared physical memory) and distributed simulation (message passing)
  • In distributed simulation: less communication would reduce total delay, may cause more rollback
  • The relative value counts: i.e. the ratio of computing time vs. communication time

• Performance models:
  • Chen's work: unify the performance model for memory effect on message distribution, rollback, GVT advances in shared memory environment using a Markov's process model

  • Gupta, Akyildiz, and Fujimoto: performance model for homogeneous and heterogeneous multiprocessor, shared memory environment
  • Kleinrock and Nicol have separately developed some bounds for the performance of Time Warp system

  • Lubachevsky, Weiss and Shwartz's work: the average time $\tau$ to complete the simulation of a system with N nodes and R events on a p-processor PRAM satisfies
    
    $$\tau ~= ~ O \left( \frac{R}{p} + \frac{R}{N} log~p \right)$$