MDB-43767: consider db in crash recovery state as db in non-terminal state

[munakoiso]

No description provided.

[munakoiso]

https://github.com/postgres/postgres/blob/57ee397953985feb95cffa65a06fcaf6a2cd5367/src/backend/tcop/backend_startup.c#L338

[aak74]

Race condition

Race condition in _rewind_from_source(): pg_rewind may start before PostgreSQL fully stops

In _rewind_from_source() /src/main.py:1271 at line 1288-1293, after calling stop_postgresql(), the code immediately proceeds to do_rewind() without waiting for PostgreSQL to actually shut down. This can cause pg_rewind to fail with "target server must be shut down cleanly" error.

Suggested Fix

Remove is_postgresql_dead parameter and always verify PostgreSQL is fully stopped before running pg_rewind:

def _rewind_from_source(self, limit, new_primary):  # removed is_postgresql_dead
    logging.info("Starting pg_rewind")
    
    # Wait for source database to be available
    if not helpers.await_for(
        lambda: not self._check_host_is_really_dead(new_primary),
        limit,
        'source database alive and ready for rewind',
    ):
        return None
    
    # Mark destructive operation in ZK
    if not self.zk.write('%s/%s/op' % (self.zk.MEMBERS_PATH, helpers.get_hostname()), 'rewind', need_lock=False):
        logging.error('Unable to save destructive op state: rewind')
        return None
    
    self.db.pgpooler('stop')
    
    # Always ensure PostgreSQL is stopped before rewind
    if self.db.get_postgresql_status() == 0:  # if process is running
        if self.stop_postgresql(timeout=limit) != 0:
            logging.error('Could not stop PostgreSQL. Will retry.')
            return None
        
        # Wait for PostgreSQL to fully stop
        if not helpers.await_for(
            lambda: self.db.get_postgresql_status() != 0,
            limit,
            'PostgreSQL fully stopped'
        ):
            logging.error('PostgreSQL did not stop in time. Will retry.')
            return None
    
    self.checks['rewind'] += 1
    if self.db.do_rewind(new_primary) != 0:
        logging.error('Error while using pg_rewind. Will retry.')
        return True
    
    # Rewind has finished successfully so we can drop its operation node
    self.zk.delete('%s/%s/op' % (self.zk.MEMBERS_PATH, helpers.get_hostname()))
    return self._attach_to_primary(new_primary, limit)

[aak74]

Race condition in _rewind_from_source(): pg_rewind may start before PostgreSQL fully stops

Added here: #138

[aak74]

LGTM