Skip to content

Commit 82ff327

Browse files
Merge pull request #959 from pfrest/next_patch
v2.11.0 Fixes & Features
2 parents 6b1cc3c + 498a990 commit 82ff327

32 files changed

Lines changed: 1091 additions & 32 deletions

‎docs/HA_SYNC.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# HA Sync
2+
3+
The REST API package supports High Availability (HA) synchronization, allowing you to automatically replicate your
4+
REST API settings from one pfSense host to one or more remote peers. This is especially useful in HA deployments where
5+
you want all nodes to share the same REST API configuration without having to configure each host manually.
6+
7+
!!! Important
8+
HA sync only synchronizes the **REST API package settings** (such as authentication methods, allowed interfaces,
9+
JWT expiration, and other options found under the REST API settings). It does not synchronize the rest of the
10+
pfSense configuration. Use pfSense's built-in HA/XMLRPC configuration sync for the base system configuration.
11+
12+
## How It Works
13+
14+
When HA sync is enabled and one or more sync hosts are defined, the source host will push its REST API settings to each
15+
configured peer whenever those settings change. Each peer receives the sync request and applies the incoming settings
16+
to its own REST API configuration.
17+
18+
## Requirements
19+
20+
Before enabling HA sync, ensure the following:
21+
22+
- The REST API package is installed on every host that will participate in synchronization.
23+
- A dedicated user exists on **every** peer defined in the sync hosts list. This user must hold the `page-all`
24+
privilege and must use the same username and password across all hosts.
25+
- Each peer is reachable from the source host over HTTPS (the webConfigurator port).
26+
- If certificate validation is enabled, each peer's webConfigurator must present a certificate trusted by the source
27+
host.
28+
29+
!!! Warning
30+
It is expected that the source host and each of its peers are as identical as possible. Changes between REST API
31+
versions or pfSense versions may make the synced configurations incompatible with the remote peers. Additionally,
32+
some settings (such as the allowed interfaces) are configuration-specific, and it is expected that those interfaces
33+
also exist on each remote peer.
34+
35+
## Configuring HA Sync on the Source Host
36+
37+
Follow these steps on the host that will send its REST API settings to peers:
38+
39+
1. In the webConfigurator, navigate to **System > REST API > Settings**.
40+
2. Enable the **HA Sync** toggle. This allows the host to both send sync requests to peers and accept incoming sync
41+
requests from peers.
42+
3. In the **HA Sync Hosts** field, enter the IP address or hostname of each remote peer that should receive this host's
43+
REST API settings. You may define up to 10 hosts.
44+
4. Set the **HA Sync Username** to the username of the account used to authenticate sync requests. This user must exist on
45+
every peer and hold the `page-all` privilege.
46+
5. Set the **HA Sync Password** to the password for the HA username. This must match on all peers.
47+
6. (Recommended) Leave **HA Sync Validate Certs** enabled to validate each peer's webConfigurator certificate. This
48+
helps mitigate Man-in-the-Middle attacks. If you disable it, certificate validation will be skipped.
49+
7. Save your changes.
50+
51+
## Configuring HA Sync on Remote Peers
52+
53+
Repeat the following on **every** remote peer listed in the source host's **HA Sync Hosts** field:
54+
55+
1. In the webConfigurator, navigate to **System > REST API > Settings**.
56+
2. Enable the **HA Sync** toggle. A peer will reject incoming sync requests by default unless the HA Sync toggle is enabled on that
57+
peer.
58+
3. Ensure the HA sync user (with the `page-all` privilege) exists on the peer and uses the same username and password
59+
as configured on the source host.
60+
4. Save your changes.
61+
62+
Once HA Sync is enabled on both the source host and all peers, any change to the source host's REST API settings will
63+
automatically be synchronized to each configured peer.
64+
65+
## Configuring HA Sync via the REST API
66+
67+
All HA sync options are also available through the REST API settings endpoint, so you can automate HA sync configuration
68+
as part of your provisioning workflow. The relevant settings include:
69+
70+
- `ha_sync` — Enables or disables HA sync on the host. Must be enabled on both the sending and receiving hosts.
71+
- `ha_sync_validate_certs` — Enables or disables certificate validation when syncing to peers.
72+
- `ha_sync_hosts` — A list of IP addresses or hostnames to sync REST API settings to.
73+
- `ha_sync_username` — The username used to authenticate sync requests. Must hold the `page-all` privilege on all peers.
74+
- `ha_sync_password` — The password for the HA sync user. Must match across all peers.

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ nav:
1313
- Working with HATEOAS: WORKING_WITH_HATEOAS.md
1414
- Common Control Parameters: COMMON_CONTROL_PARAMETERS.md
1515
- GraphQL: GRAPHQL.md
16+
- HA Sync: HA_SYNC.md
1617
- Securing API Access: SECURING_API_ACCESS.md
1718
- Limitations & FAQs: LIMITATIONS_AND_FAQS.md
1819
- API Reference: https://pfrest.org/api-docs/

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Caches/PrivilegesCache.inc‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,13 @@ class PrivilegesCache extends Cache {
1717
*/
1818
public function get_data_to_cache(): array {
1919
# Use a variable to keep track of the privileges associated with this endpoint
20-
$privs = [];
20+
$privs = [
21+
'page-system-restapi-documentation' => [
22+
'name' => 'WebCfg - System: RESTAPI Documentation',
23+
'descr' => 'Allow access to the /api/v2/documentation page.',
24+
'match' => ['api/v2/documentation*'],
25+
],
26+
];
2127

2228
# Obtain privileges for each Endpoint class
2329
foreach (get_classes_from_namespace('\\RESTAPI\\Endpoints\\') as $endpoint_class) {

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Core/Field.inc‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ class Field {
8383
* Make this a required field. If false, a valid $default, $default_callable or $allow_null must be set.
8484
* @param bool $unique
8585
* If true, $value must be unique among all parent Model objects (only for parent Models with $many set to true).
86+
* @param bool $skip_unique_validation
87+
* If true, fields marked as unique will skip enforcement of the unique constraint. This is necessary for
88+
* fields that use $unique for schema or indexing purposes, but do not need the constraint enforced.
8689
* @param mixed $default
8790
* The default $value for this field if none is assigned. The default specified here should be static. If the
8891
* default for this field is dynamic in nature, use a $default_callable instead.
@@ -136,6 +139,7 @@ class Field {
136139
public string $type = '',
137140
public bool $required = false,
138141
public bool $unique = false,
142+
public bool $skip_unique_validation = false,
139143
public mixed $default = null,
140144
public string $default_callable = '',
141145
public array $choices = [],
@@ -725,8 +729,8 @@ class Field {
725729
* @throws ServerError When $unique is enabled, but the parent model $context does not have $many enabled
726730
*/
727731
private function check_field_unique(mixed $value, ?ModelSet $modelset = null): void {
728-
# Only check this if `unique` is enabled
729-
if ($this->unique) {
732+
# Only check this if `unique` is enabled and the unique constraint is enforced
733+
if ($this->unique and !$this->skip_unique_validation) {
730734
# Throw an error if unique is set but the model context is not a 'many' model.
731735
if (!$this->context->many) {
732736
throw new ServerError(

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Core/Model.inc‎

Lines changed: 34 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1870,6 +1870,17 @@ class Model {
18701870
$this->apply();
18711871
}
18721872

1873+
/**
1874+
* Prepares representation data before validating a `replace_all()` request, including dry runs.
1875+
* Override this method to supply operation-specific defaults without changing other CRUD actions.
1876+
* @param array $data The requested Model representations.
1877+
* @param ModelSet $initial_objects The existing Model objects before replacement.
1878+
* @return array The prepared Model representations to validate and replace.
1879+
*/
1880+
protected function pre_validate_replace_all(array $data, ModelSet $initial_objects): array {
1881+
return $data;
1882+
}
1883+
18731884
/**
18741885
* Performs steps and processes required to apply changes after the `replace_all()` method runs. This can be
18751886
* overridden to add custom steps to apply after replace all actions only. Defaults to simply calling `apply()`.
@@ -2446,6 +2457,9 @@ class Model {
24462457
# if needed.
24472458
$initial_objects = $this->query(parent_id: $this->parent_id);
24482459

2460+
# Prepare operation-specific defaults before validation or any configuration changes
2461+
$data = $this->pre_validate_replace_all($data, $initial_objects);
2462+
24492463
# Obtain any Models that are deemed protected to ensure they do not removed in the next step.
24502464
$new_objects = new ModelSet();
24512465
if ($this->protected_model_query) {
@@ -2456,21 +2470,28 @@ class Model {
24562470
# do not mutate or persist the configuration. This still allows clients to detect duplicate or invalid
24572471
# entries before applying the replacement.
24582472
if ($this->dry_run) {
2459-
foreach ($data as $representation_object) {
2460-
unset($representation_object['id']);
2461-
$model_object = new $this(data: $representation_object, skip_init: true, client: $this->client);
2462-
# Assign a synthetic ID for uniqueness comparisons during validation
2463-
$model_object->id = count($new_objects->model_objects);
2464-
$model_object->validate(modelset: $new_objects);
2465-
$new_objects->model_objects[] = $model_object;
2473+
# Use the root signature so self-references resolve against the simulated replacement set
2474+
$new_objects = new ModelSet(model_objects: $new_objects->model_objects, signature: $this->get_class_fqn());
24662475

2467-
# In the event that the model can reference itself (e.g. FirewallAlias), ensure the model cache is
2468-
# updated so ForeignModelField can accurately find referenced objects.
2469-
self::get_model_cache()::cache_modelset($new_objects);
2470-
}
2476+
try {
2477+
foreach ($data as $representation_object) {
2478+
# Discard stale queries and indexes before caching the growing replacement set
2479+
self::clear_model_cache();
2480+
self::get_model_cache()::cache_modelset($new_objects);
2481+
2482+
unset($representation_object['id']);
2483+
$model_object = new $this(data: $representation_object, skip_init: true, client: $this->client);
2484+
# Assign a synthetic ID for uniqueness comparisons during validation
2485+
$model_object->id = count($new_objects->model_objects);
2486+
$model_object->validate(modelset: $new_objects);
2487+
$new_objects->model_objects[] = $model_object;
2488+
}
24712489

2472-
self::clear_model_cache();
2473-
return $new_objects;
2490+
return $new_objects;
2491+
} finally {
2492+
# Never retain simulated objects, including when validation throws an error
2493+
self::clear_model_cache();
2494+
}
24742495
}
24752496

24762497
# Before beginning validation, empty the stored values in the config path. Retain protected Models (if any).

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/Base64Field.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ class Base64Field extends Field {
1818
* @param bool $required If `true`, this field is required to have a value at all times.
1919
* @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
2020
* option requires the Model $context to be set AND the Model $context must have a `config_path` set.
21+
* @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
2122
* @param mixed|null $default Assign a default string value to assign this Field if no value is present.
2223
* @param string $default_callable Defines a callable method that should be called to populate the default value
2324
* for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -72,6 +73,7 @@ class Base64Field extends Field {
7273
public function __construct(
7374
bool $required = false,
7475
bool $unique = false,
76+
bool $skip_unique_validation = false,
7577
mixed $default = null,
7678
string $default_callable = '',
7779
array $choices = [],
@@ -101,6 +103,7 @@ class Base64Field extends Field {
101103
type: 'string',
102104
required: $required,
103105
unique: $unique,
106+
skip_unique_validation: $skip_unique_validation,
104107
default: $default,
105108
default_callable: $default_callable,
106109
choices: $choices,

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/BooleanField.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ class BooleanField extends RESTAPI\Core\Field {
2727
* @param bool $required If `true`, this field is required to have a value at all times.
2828
* @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
2929
* option requires the Model $context to be set AND the Model $context must have a `config_path` set.
30+
* @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
3031
* @param mixed|null $default Assign a default string value to assign this Field if no value is present.
3132
* @param string $default_callable Defines a callable method that should be called to populate the default value
3233
* for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -70,6 +71,7 @@ class BooleanField extends RESTAPI\Core\Field {
7071
public function __construct(
7172
bool $required = false,
7273
bool $unique = false,
74+
bool $skip_unique_validation = false,
7375
mixed $default = null,
7476
string $default_callable = '',
7577
bool $allow_empty = false,
@@ -95,6 +97,7 @@ class BooleanField extends RESTAPI\Core\Field {
9597
type: 'boolean',
9698
required: $required,
9799
unique: $unique,
100+
skip_unique_validation: $skip_unique_validation,
98101
default: $default,
99102
default_callable: $default_callable,
100103
allow_empty: $allow_empty,

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/DateTimeField.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ class DateTimeField extends Field {
1919
* @param bool $required If `true`, this field is required to have a value at all times.
2020
* @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
2121
* option requires the Model $context to be set AND the Model $context must have a `config_path` set.
22+
* @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
2223
* @param mixed|null $default Assign a default string value to assign this Field if no value is present.
2324
* @param string $default_callable Defines a callable method that should be called to populate the default value
2425
* for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -78,6 +79,7 @@ class DateTimeField extends Field {
7879
public function __construct(
7980
bool $required = false,
8081
bool $unique = false,
82+
bool $skip_unique_validation = false,
8183
mixed $default = null,
8284
string $default_callable = '',
8385
array $choices = [],
@@ -117,6 +119,7 @@ class DateTimeField extends Field {
117119
type: 'string',
118120
required: $required,
119121
unique: $unique,
122+
skip_unique_validation: $skip_unique_validation,
120123
default: $default,
121124
default_callable: $default_callable,
122125
choices: $choices,

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/FilterAddressField.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ class FilterAddressField extends InterfaceField {
2222
* @param bool $required If `true`, this field is required to have a value at all times.
2323
* @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
2424
* option requires the Model $context to be set AND the Model $context must have a `config_path` set.
25+
* @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
2526
* @param mixed|null $default Assign a default string value to assign this Field if no value is present.
2627
* @param string $default_callable Defines a callable method that should be called to populate the default value
2728
* for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -90,6 +91,7 @@ class FilterAddressField extends InterfaceField {
9091
public function __construct(
9192
bool $required = false,
9293
bool $unique = false,
94+
bool $skip_unique_validation = false,
9395
mixed $default = null,
9496
string $default_callable = '',
9597
public bool $allow_ipaddr = true,
@@ -128,6 +130,7 @@ class FilterAddressField extends InterfaceField {
128130
parent::__construct(
129131
required: $required,
130132
unique: $unique,
133+
skip_unique_validation: $skip_unique_validation,
131134
default: $default,
132135
default_callable: $default_callable,
133136
interface_query: $interface_query,

‎pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/FloatField.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ class FloatField extends Field {
1616
* @param bool $required If `true`, this field is required to have a value at all times.
1717
* @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
1818
* option requires the Model $context to be set AND the Model $context must have a `config_path` set.
19+
* @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
1920
* @param mixed|null $default Assign a default string value to assign this Field if no value is present.
2021
* @param string $default_callable Defines a callable method that should be called to populate the default value
2122
* for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -70,6 +71,7 @@ class FloatField extends Field {
7071
public function __construct(
7172
bool $required = false,
7273
bool $unique = false,
74+
bool $skip_unique_validation = false,
7375
mixed $default = null,
7476
string $default_callable = '',
7577
array $choices = [],
@@ -98,6 +100,7 @@ class FloatField extends Field {
98100
type: 'double', // gettype() identifies floats as doubles
99101
required: $required,
100102
unique: $unique,
103+
skip_unique_validation: $skip_unique_validation,
101104
default: $default,
102105
default_callable: $default_callable,
103106
choices: $choices,

0 commit comments

Comments
 (0)