Acacia I2C, Serial and Timer + new Python tooling by omeh-a · Pull Request #751 · au-ts/sddf

omeh-a · 2026-07-08T00:56:21Z

This PR introduces Acacia to the sDDF and reworks all of our meta tooling around it.

Instead of storing our sDDF driver generation code in sdf_gen, we now store it locally in ./acacia_sddf/driver_class.py. Common logic is in ./acacia_sddf/sddf.py
tools/meta is removed and remaining Python like board.py is moved to ./acacia_sddf. This is to allow all of the Python in the sDDF to sit in one module that can be imported all in one blast. I moved acacia_sddf to the root of the repository to reflect its significance, as this code is in fact critical to assemble anything in the sDDF (as opposed to being a "two generic subdirectories deep tool")
board.py has been changed to get rid of some indirection, we now store compatible strings there for sanity.
All config.jsons are redundant and replaced with new functionality in driver_manifest.py. This file offers an API to let this same information be stored globally, and I intend for this to be used by each driver_class.py ... e.g. in i2c.py

# meson
add_driver_config(
    "meson",
    sDDFDriverConfig(
        compatible="amlogic,meson-axg-i2c",
        regions=[DTSRegion("regs", "rw", 4096, 0)],
        irqs=[DTSIRQ(0), DTSIRQ(1)],
    ),
)

# opentitan
add_driver_config(
    "opentitan",
    sDDFDriverConfig(
        compatible="eth,i2c",
        regions=[DTSRegion("regs", "rw", 4096, 0)],
        irqs=[DTSIRQ(4), DTSIRQ(0), DTSIRQ(1), DTSIRQ(7), DTSIRQ(9)],
    ),
)

... i.e. we store the old config.json in i2c.py.

There's not too much to see otherwise besides the new Acacia subsystem generation in i2c.py, serial.py and timer.py. Please let me know what you think about these.

I am leaving this PR as a draft until we have merged Acacia, just in case API changes occur.

I've implemented these three classes as an exemplar for Acacia usage in advance of porting the other classes by others. These three are also mutually dependent and it isn't possible to do i2c without the other two.

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au> Timer tests out with acacia Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au> Serial tests out with Acacia Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au> Refactor: acacia_sddf is now a python module that inherits all subfiles, sDDF itself is a python module for import! Needed for SDK Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au> Fix issues with i2c.py - maps were swapped. Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au> Further fixes to keep up with Acacia PR request changes Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

wom-bat · 2026-07-08T05:12:39Z

+    blk: Optional[DriverDouble] = DriverDouble(None, None)
+    partition: int = 0
+    baud_rate: Optional[int] = None
+


How does this work when you have multiple ethernet devices (of the same or different kinds) in the same board? I think these should be lists of DriverDouble

It didn't before these changes, from tools/meta/board.py it was reasonably similar. @Courtney3141 might need to comment on what the Firewall does, which I think uses a different board definition.

In general I think we mostly assume we have "single" instances of the driver subsystems/components. There are plenty of cases where it makes sense for systems to have multiple different block drivers, or i2c, or serial, and even multiple connections per client.

I think for now we should keep assuming that.

Maybe what we should do instead is that when you construct a system, by default what should happen is we iterate over the DTS file and instantiate a driver for every compatible string in the DTS we know about.

We can have a user API that has a stable naming/ordering scheme and then you ask for the Network subsystem "manager" to connect your client to e.g. ethernet device 0 or 1, or ethernet device at this PCI bus location, or identified by this PCI vendor+device, or at this DTS location. Most of our examples can then get away with asking for device 0 (because, maybe say, we use the DTS chosen node in the DTS we provide to force it to be the right one).

Then maybe users of our build system can construct systems in which you have connections, and if a particular block driver has no connections then we purge it from the system. (For clock drivers, maybe not, though you would assume there would be a connection between them and device drivers depending on them, so it could probably follow the same idea).

This makes sense in the LionsOS context too, really.

wom-bat · 2026-07-08T05:14:00Z

+
+# Keep this list in alphabetical order by board name
+# TODO: convert to Dictionary
+BOARDS: List[Board] = [


I personally would prefer splitting these out one per supported board, then just importing the one you want.

(I know the old sdfgen tooling did it this way)

Then examples need to import every board anyway?

wom-bat · 2026-07-08T05:16:16Z

+from .sddf import sDDFDriverClass, DeviceResourcesFactory, RegionResourceFactory
+from collections import defaultdict
+from typing import List, Dict, Type, Union, Optional
+


rearrange imports to put standard ones first

Also reframe from multiple imports on one line.

wom-bat · 2026-07-08T05:20:00Z

+
+
+# pulp
+add_driver_config(


I reckon these should be in per-driver files imported rather than enumerated here. So that the tooling can be used with third-party drivers without changing this file.

Well, you can already, see 'add_driver_config' you can import that and call it. I think there needs to be an global registry of drivers in some sense otherwise your meta program needs to duplicate driver selection logic.

Though it might be better to follow the "driver finder library" approach or something instead of doing what Linux calls the midlayer mistake. But then maybe that doesn't work with my proposed solution of creating drivers for every found DTS node where a compatible matches?

wom-bat · 2026-07-08T05:21:16Z


 SUPPORTED_BOARDS := \
 		odroidc4 \
+		maaxboard \


separate change, different PR

wom-bat · 2026-07-08T05:35:56Z

    board = next(filter(lambda b: b.name == args.board, BOARDS))
+    if board.arch != x86_64:
+        dtb = DeviceTreeBlob(args.dtb)
+    else:


An alternative would be to fudge up a DTS for the X86 systems we use. Linux X86 can use a DTB as an additional source of information (mainly to get the PCI and IOAPIC info)

Something like that isn't a bad idea... for what it's worth: this check is redundant. If there's no DTB given, this creates an empty dtb which is recognised by Acacia as being an x86 system

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

midnightveil · 2026-08-04T04:59:37Z

@@ -0,0 +1,8 @@
+# Copyright 2026, UNSW


I don't think this should be under acacia_sddf. It's not really clear what acacia is for a user. I think the timer/serial classes should either live in the appropriate serial/ or timer/ directory and helpers under tools/meta or tools/acacia.

There is a blunt reason for this: it means that we only need one Python import and it means that each driver class doesn't need to abuse importlib to find the others. I really think it's fine to have end users learn what Acacia is, given that they have to install it to use the sDDF. I did it as you suggest initially and it made things horrible. I'm really not eager to change this.

I don't think having to set up PYTHONPATH (which you can do entirely in the Makefile) is that bad. If you want just one import you can still do that without separating the components from the rest of the components, you just have some path-mangling.

The other thing is to consider external users: LionsOS, for instance, could just do "import sddf.timer" if it's located in the timer folder, whereas with this way you need to either do "import sddf.acacia_sddf.timer" or have top level exports (not necessary if your project structure follows python module structure, no need for init.py etc). They'd have to put sDDF in their PythonPath but that's fine and reasonable.

tldr; no, I disagree.

Users of anything will be using acacia anyway though? How can a user of acacia who must be using it in their metaprogram not already know what acacia is? Making the entire sDDF tree a python module just complicates things. The only functional diff from what you're describing is whether the module is called sddf or acacia_sddf. We can rename acacia_sddf to just sddf if it matters ... I personally think it's inconsequential, but I also really think it's worse for code organisation to scatter the python through the source tree.

Sidenote: I don't think we will need to import sddf.acacia_sddf.blah either? We can do this exactly as it is done in the metaprogram and not make a top level sDDF module

midnightveil · 2026-08-04T05:12:07Z

+    blk: Optional[DriverDouble] = DriverDouble(None, None)
+    partition: int = 0
+    baud_rate: Optional[int] = None
+


It didn't before these changes, from tools/meta/board.py it was reasonably similar. @Courtney3141 might need to comment on what the Firewall does, which I think uses a different board definition.

In general I think we mostly assume we have "single" instances of the driver subsystems/components. There are plenty of cases where it makes sense for systems to have multiple different block drivers, or i2c, or serial, and even multiple connections per client.

midnightveil · 2026-08-04T05:13:14Z

+
+# Keep this list in alphabetical order by board name
+# TODO: convert to Dictionary
+BOARDS: List[Board] = [


Then examples need to import every board anyway?

midnightveil · 2026-08-04T05:23:20Z

+    blk: Optional[DriverDouble] = DriverDouble(None, None)
+    partition: int = 0
+    baud_rate: Optional[int] = None
+


I think for now we should keep assuming that.

Maybe what we should do instead is that when you construct a system, by default what should happen is we iterate over the DTS file and instantiate a driver for every compatible string in the DTS we know about.

We can have a user API that has a stable naming/ordering scheme and then you ask for the Network subsystem "manager" to connect your client to e.g. ethernet device 0 or 1, or ethernet device at this PCI bus location, or identified by this PCI vendor+device, or at this DTS location. Most of our examples can then get away with asking for device 0 (because, maybe say, we use the DTS chosen node in the DTS we provide to force it to be the right one).

Then maybe users of our build system can construct systems in which you have connections, and if a particular block driver has no connections then we purge it from the system. (For clock drivers, maybe not, though you would assume there would be a connection between them and device drivers depending on them, so it could probably follow the same idea).

This makes sense in the LionsOS context too, really.

midnightveil · 2026-08-04T05:57:29Z

+
+
+# pulp
+add_driver_config(


Well, you can already, see 'add_driver_config' you can import that and call it. I think there needs to be an global registry of drivers in some sense otherwise your meta program needs to duplicate driver selection logic.

Though it might be better to follow the "driver finder library" approach or something instead of doing what Linux calls the midlayer mistake. But then maybe that doesn't work with my proposed solution of creating drivers for every found DTS node where a compatible matches?

midnightveil · 2026-08-04T05:58:38Z

-        f.write(sdf.render())
+    out_file = f"{output_dir}/{sdf_file}"
+    sdf.make_config_structs()
+    print(f"Saving to {out_file}")


midnightveil · 2026-08-04T05:59:03Z

-        f.write(sdf.render())
+    out_file = f"{output_dir}/{sdf_file}"
+    sdf.make_config_structs()
+    print(f"Saving to {out_file}")


midnightveil · 2026-08-04T06:08:57Z

+        assert type(self.irqs) is list
+
+
+class __sDDFDriverManifest:


If anything I feel like this PR should just transliterate the exact same functionality from the zig sdfgen classes to Python, so that the difference is extremely minimal limited to a few type changes and import differences. Then we can discuss changing the API exposed in meta.py files at a later point in time to improve them wholistically.

These changes were necessary due to acacia working differently than sdf-gen. There's no real way around this without basically just putting the same functionality elsewhere in the python that is added, and indeed this is doing something similarly to what sdf_gen did by scanning the sddf for config.jsons and keeping a register of them

Not all of the changes were necessary. Perhaps moving away from config.json is better. Is it definitely? IDK. But it's a separate change that is logically distinct from a Zig -> Python rewrite.

We can't just do a zig to python rewrite when we need to move all the changes that were formerly in a different tool, in a different language, to here. Again, this is just doing something similar to what sdf_gen did.

See sddf.zig ... this is a replacement for these

var drivers: std.array_list.Managed(Config.Driver) = undefined; var classes: std.array_list.Managed(Config.DeviceClass) = undefined;

We can obviously change this in the future. But this bookkeeping isn't new.

(I can add searching for config.jsons if we want, just simpler calling direct for now..)

Not sure if this was considered and then decided against but can you avoid all the add_driver_config repetition and use the json by doing something like this:

Make the sDDFDriverClass class use the DT path to find the DT node like it does atm, then can it retrieve the compatible strings and provide this to the sDDFDriverManifest. Then the sDDFDriverManifest selects the json config from the compatible strings and parses the config into the form equivalent to what all the add_driver_config function calls are doing atm? I guess you would also provide the driver class name so the manifest knows which directory to search.

Make the sDDFDriverClass class use the DT path to find the DT node like it does atm, then can it retrieve the compatible strings and provide this to the sDDFDriverManifest. Then the sDDFDriverManifest selects the json config from the compatible strings and parses the config into the form equivalent to what all the add_driver_config function calls are doing atm? I guess you would also provide the driver class name so the manifest knows which directory to search.

The main thing is that we will need to rejig this all soon anyway. Fundamentally the sdf_gen model isn't really any good for handling dependencies or per-board deviations. The goal here for me was to basically do the same thing minimally for now

Signed-off-by: Courtney Darville <courtneydarville94@outlook.com>

cazb2 · 2026-08-08T02:21:36Z

+
+
+@dataclass
+class Board:


Can we remove the board.py indirection? This seems to duplicate information already present in the DTB which is never ideal since a human has to do this and is likely to make mistakes (such as the zcu paddr being wrong previously).

This is probably a consideration independent of this PR.

Per other comment I replied to, basically yes we can, and we probably should, but that's something that is beyond the scope of just adding Acacia

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

… update classes to respect this. Minor changes to api; got rid of (perilous) manual call to create resources Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

omeh-a · 2026-08-27T23:59:28Z

Closing this draft PR to re-open smaller PRs for individual reviewers, as discussed in systems meeting.

midnightveil · 2026-08-28T00:06:01Z

Can you just update this PR instead for the "baseline" things? Otherwise we lose all the PR comments?

omeh-a · 2026-08-28T00:24:17Z

Can you just update this PR instead for the "baseline" things? Otherwise we lose all the PR comments?

Per discussion in the meeting, this doesn't work very well. The GitHub UI has been broken for weeks and conversations don't show properly across separate reviews. Given that we need to loop in several more reviewers, working here is going to become a disaster immediately. This PR existed when we originally planned to have people other than me complete most of the driver classes, but that didn't end up working, and we thus need to adapt.

Courtney3141 · 2026-08-28T03:27:09Z

In the interest of not losing the history/discussions in this PR (which I imagine will be quite relevant/helpful when reviewing the code in the future), I don't think it's a bad idea to use this PR for the base components (#775), and split out the other subsystems into separate PRs.

Personally when I review, I find it helpful to read through previous discussions, even if they have now been resolved. Additionally, as @midnightveil mentioned, opening a new PR makes it easier to miss an unaddressed issue accidentally.

The GitHub UI has been broken for weeks and conversations don't show properly across separate reviews

Agreed that the UI has been buggy, but having some record of our discussions is still better than nothing IMO.

Given that we need to loop in several more reviewers, working here is going to become a disaster immediately

We will only need to loop in whoever is responsible for reviewing #775, and I think having a record of previous discussion will still be helpful. If the large number of old comments does become a hindrance we can open a new PR then, but otherwise I think it's best practice to keep design discussions in a single PR (I would prefer not to have a repeat of the timer re-design situation where I needed to jump between two PRs to understand the background for the design). Longer term it is also useful to keep design/review discussions in one place if possible.

If we use this PR for the base components, then we only need to worry about keeping track of serial/timer/i2c issues raised in this PR (as opposed to creating issues for all unresolved discussions).

From the discussion in #775:

Pretty much all of what is outstanding that is pertinent are issues that we should not be solving monolithically before introducing Acacia

This may be the case (that some issues raised will not be solved by the merging of this PR), but I don't think it follows that this necessitates us creating a new PR without those discussions. In fact, it is not uncommon that PR discussions result in issues we agree to solve later.

We really need to get it merged into the sDDF now before we start fixing the whole family of issues related to the current way we handle driver config / dependencies

This may be true, but I don't see why this means we need to create a new PR - it just means we need to agree that we will hold off on fixing everything immediately. This can be done without opening a new PR obscuring previously raised issues.

All concerns there have been discussed in person, as was the decision to make a new PR

I know we definitely agreed to split up the PR into subsystems etc, but I don't think we decided we needed to close the original PR.

Ultimately, I think it would be nice if we could use this PR for the base components. Then when it is merged, the team's discussions that lead to the ultimate design will all be in one place (#751). If we use #775, it is not immediately clear that a great deal of the relevant conversation is contained in the now closed #751.

omeh-a · 2026-08-28T04:09:36Z

Ultimately, I think it would be nice if we could use this PR for the base components. Then when it is merged, the team's discussions that lead to the ultimate design will all be in one place (#751). If we use #775, it is not immediately clear that a great deal of the relevant conversation is contained in the now closed #751.

If that's the case, I guess we should pull all the changes from the other parts out of here? It just will really mangle the conversation history, hence moving things out. Most of the discussion here really pertains to how to handle using the DTB better and getting rid of board.py, both of which are things where we can refer back to using this closed issue. Most of the comments here actually are related to the driver classes and not the base, which makes this sticky... also: most of the comments are stale as they were made while all of these classes were very drafty. Is there anything in particular here w.r.t. the base components that is super important besides the above? If so, I'm inclined to just make new issues in the new PR that link to the discussion here

omeh-a · 2026-08-28T04:34:50Z

On second thought, I guess we just leave this here then. It's more of an issue for the timer, i2c, etc. I'm going to reopen this for now and we should figure out what to do

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Courtney3141 · 2026-08-28T07:54:43Z

Most of the discussion here really pertains to how to handle using the DTB better and getting rid of board.py, both of which are things where we can refer back to using this closed issue.

If that's true, we could use this PR for base components (if that handles DTB/board)

Most of the comments here actually are related to the driver classes and not the base, which makes this sticky... also: most of the comments are stale as they were made while all of these classes were very drafty.

That's okay, we can always mark them as resolved (if they are), and raise the others as issues.

Is there anything in particular here w.r.t. the base components that is super important besides the above? If so, I'm inclined to just make new issues in the new PR that link to the discussion here

I can't speak to the discussion myself as I haven't read it yet, but I would rather keep this discussion as is, and raise the non-relevant parts as issues separately, than have to raise everything. And as I said, when I review things I pretty much always read all the previous discussions, as it helps me get a sense of other's opinions/things I missed/better understand design choices etc. So I think it's best not to close and re-open PRs unless the code has undergone some substantial change (even then I am not sure if I have ever closed/reopened the same PR).

omeh-a added 3 commits July 3, 2026 09:20

Updated ina219 and bus scan i2c examples to use acacia

a55c47c

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Lint Python with Acacia additions

013449b

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

omeh-a added drivers Issues pertaining to driver code for a device class driver-examples Issues related to examples for drivers for a target serial timer i2c labels Jul 8, 2026

wom-bat requested changes Jul 8, 2026

View reviewed changes

Update signatures to put SDF first for Acacia

6d06dfb

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

omeh-a force-pushed the sdfgenpy branch from 370e7a7 to 6d06dfb Compare July 12, 2026 23:39

omeh-a added 2 commits July 13, 2026 10:09

Extract per-board / x86 metaprogram hacks to class.py

de0f706

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Add I2CAddress type and I2C address decoding for I2C class

55b8d7b

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Courtney3141 mentioned this pull request Jul 25, 2026

Update config structs module to use a parser to create a tree from dwarf dump output au-ts/microkit_acacia#22

Merged

midnightveil requested changes Aug 4, 2026

View reviewed changes

midnightveil reviewed Aug 4, 2026

View reviewed changes

omeh-a force-pushed the sdfgenpy branch from a95f14c to 1d5b5c5 Compare August 6, 2026 05:58

Courtney3141 added 2 commits August 7, 2026 15:45

Increase magic length to accomodate null terminator

e3b63d2

Signed-off-by: Courtney Darville <courtneydarville94@outlook.com>

Update the order of arguments in config structs

d440357

Signed-off-by: Courtney Darville <courtneydarville94@outlook.com>

cazb2 reviewed Aug 8, 2026

View reviewed changes

omeh-a added 5 commits August 10, 2026 14:11

Fix evil sneaky bug which sends offset to segment name

a596f43

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Refactor sDDFDeviceClass to initialise configstructs at correct time,…

3db0a83

… update classes to respect this. Minor changes to api; got rid of (perilous) manual call to create resources Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Added blk support using acacia

b3efcfb

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Add informative print for DT indices outside of DTB range

0c39f19

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

Add ethernet Acacia module (WIP)

e1d76d2

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

omeh-a force-pushed the sdfgenpy branch from 1d5b5c5 to e1d76d2 Compare August 27, 2026 23:49

omeh-a closed this Aug 27, 2026

omeh-a mentioned this pull request Aug 28, 2026

Add base components for Acacia #775

Closed

omeh-a reopened this Aug 28, 2026

omeh-a force-pushed the sdfgenpy branch from 6f397a2 to ec5470f Compare August 28, 2026 04:56

bring back changes from side PRs

f57aec1

Signed-off-by: Lesley Rossouw <lesley.rossouw@unsw.edu.au>

omeh-a force-pushed the sdfgenpy branch from ec5470f to f57aec1 Compare August 28, 2026 04:58

Uh oh!

Conversation

omeh-a commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omeh-a Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omeh-a commented Aug 27, 2026

Uh oh!

midnightveil commented Aug 28, 2026

omeh-a commented Jul 8, 2026 •

edited

Loading

omeh-a Aug 5, 2026 •

edited

Loading