/*
   Copyright (c) 2003, 2024, Oracle and/or its affiliates.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License, version 2.0,
   as published by the Free Software Foundation.

   This program is designed to work with certain software (including
   but not limited to OpenSSL) that is licensed under separate terms,
   as designated in a particular file or component or in included license
   documentation.  The authors of MySQL hereby grant you an additional
   permission to link the program and your derivative works with the
   separately licensed software that they have either included with
   the program or referenced in the documentation.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License, version 2.0, for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301  USA
*/

--- Start phase 1 - Qmgr -------------------------------------------

1) Set timer 1 - TimeToWaitAlive

2) Send CM_REGREQ to all connected(and connecting) nodes

3) Wait until -
a) The precident answers CM_REGCONF
b) All nodes has answered and I'm the candidate -> election won
c) 30s has passed and I'm the candidate -> election won
d) TimeToWaitAlive has passed -> Failure to start

When receiving CM_REGCONF
4) Send CM_NODEINFOREQ to all connected(and connecting) nodes 
   reported in CM_REGCONF

5) Wait until -
a) All CM_NODEINFO_CONF has arrived
b) TimeToWaitAlive has passed -> Failure to start

6) Send CM_ACKADD to president

7) Wait until - 
a) Receive CM_ADD(CommitNew) from president -> I'm in the qmgr cluster
b) TimeToWaitAlive has passed -> Failure to start

NOTE:
30s is hardcoded in 3c.
TimeToWaitAlive should be atleast X sec greater than 30s. i.e. 30+X sec
to support "partial starts"

NOTE:
In 3b, a more correct number (instead of all) would be
N-NG+1 where N is #nodes and NG is #node groups = (N/R where R is # replicas)
But Qmgr has no notion about node groups or replicas

--- Start phase X - Qmgr -------------------------------------------

President - When accepting a CM_REGREQ
1) Send CM_REGCONF to starting node
2) Send CM_ADD(Prepare) to all started nodes + starting node
3) Send CM_ADD(AddCommit) to all started nodes
4) Send CM_ADD(CommitNew) to starting node

Cluster participant -
1) Wait for both CM_NODEINFOREQ from starting and CM_ADD(Prepare) from pres.
2) Send CM_ACKADD(Prepare)
3) Wait for CM_ADD(AddCommit) from president
4) Send CM_ACKADD(AddCommit) 
 
--- Start phase 2 - NdbCntr ----------------------------------------

- Use same TimeToWaitAliveTimer

1) Check sysfile (DIH_RESTART_REQ)
2) Read nodes (from Qmgr) P = qmgr president

3) Send CNTR_START_REQ to cntr(P)
   including info in DIH_RESTART_REF/CONF

4) Wait until -
b) Receiving CNTR_START_CONF -> continue
b) Receiving CNTR_START_REF -> P = node specified in REF, goto 3
c) TimeToWaitAlive has passed -> Failure to start

4) Run ndb-startphase 1

--
Initial start/System restart NdbCntr (on qmgr president node)

1) Wait until -
a) Receiving CNTR_START_REQ with GCI > than own GCI
   send CNTR_START_REF to all waiting nodes
b) Receiving all CNTR_START_REQ (for all defined nodes)
c) TimeToWait has passed and partition win
d) TimeToWait has passed and partitioning 
   and configuration "start with partition" = true

2) Send CNTR_START_CONF to all nodes "with filesystem"

3) Wait until -
   Receiving CNTR_START_REP for all starting nodes

4) Start waiting nodes (if any)

NOTE:
1c) Partition win = 1 node in each node group and 1 full node group
1d) Pattitioning = at least 1 node in each node group
-- 
Running NdbCntr

When receiving CNTR_MASTER_REQ
1) If I'm not master send CNTR_MASTER_REF (including master node id)
2) If I'm master
   Coordinate parallell node restarts
   send CNTR_MASTER_CONF (node restart)
