Automation Crawler & Desktop Automation — Architecture Decision Record
1. Problem Definition
The system is intended to build a crawler that can discover all meaningful user flows through an application.
The initial scope was web applications:
Crawl a website.
Discover screens.
Map the contents of each screen.
Identify interactive elements.
Discover possible actions.
Execute actions.
Detect resulting states.
Build a graph of screens/states and transitions.
Generate meaningful user flows from that graph.
The architecture was then extended to support desktop automation as well as web automation.
The final direction is therefore broader:
Build a general-purpose interaction graph and automation engine, where web, desktop, and future environments are execution adapters underneath a common Go kernel.
2. Core Product Concept
The product should not fundamentally be thought of as a "web crawler."
It is better understood as:
State-space explorer + screen mapper + action discoverer + workflow graph builder.
The primary output is an interaction graph.
Example:
Homepage
│
├── click Login ───────> Login
│ │
│ ├── submit ───> Dashboard
│ │
│ └── forgot password ──> Reset Password
│
├── click Search ──────> Search
│ │
│ └── search ──> Results
│
└── click Pricing ─────> Pricing
A user flow is then a path through this graph.
For example:
Homepage
→ Search
→ Search Results
→ Product
→ Cart
→ Checkout
can be interpreted as:
User purchases a product.
3. First Architectural Decision
The crawler should be divided into four conceptual problems:
1. Screen / State Discovery
2. Screen Mapping
3. Action Discovery
4. Transition Discovery
These should not be implemented as one giant crawler loop.
Screen discovery
Determine:
What screens/states exist?
Screen mapping
Determine:
What exists on this screen?
Action discovery
Determine:
What can the user do?
Transition discovery
Determine:
What happens when the user performs an action?
This separation makes the system easier to test, extend, and debug.
4. State Is More Important Than URL
A URL is not sufficient to identify a state.
For example:
/cart
could represent:
/cart
├── empty
├── item added
├── coupon applied
├── checkout modal open
└── authenticated checkout
Therefore the system needs an explicit state model.
Conceptually:
type BrowserState struct {
URL string
Cookies []Cookie
LocalStorage map[string]string
SessionStorage map[string]string
ScreenshotHash string
DOMHash string
AuthState string
Overlays []string
SelectedElements map[string]string
FormState map[string]string
}
The final architecture generalizes this into a runtime-independent state model.
5. State Fingerprinting
The crawler must prevent repeated exploration of the same state.
A state fingerprint can combine:
normalized URL
+
DOM structure hash
+
accessibility tree hash
+
visual perceptual hash
+
relevant storage/session state
+
authentication state
+
relevant UI state
But exact equality is not always appropriate.
State comparison should support similarity:
URL same?
DOM similarity?
Visual similarity?
Same important controls?
Same auth state?
Same relevant UI state?
Then:
if similar(existing_state, new_state):
merge states
else:
create new state
This prevents loops such as:
A → B → B → B → B → ...
6. Exploration Algorithm
The core crawler is fundamentally a graph search.
Breadth-first search was identified as a good initial approach because it naturally discovers relatively short paths.
Conceptually:
START
↓
Open initial surface
↓
Observe
↓
Map screen
↓
Discover actions
↓
Pick unexplored action
↓
Execute action
↓
Observe resulting state
↓
State already known?
├── YES → record transition
└── NO → add state
↓
map state
↓
discover actions
↓
repeat
Pseudo-code:
queue := []ExplorationTask{initialTask}
for len(queue) > 0 {
task := queue.Pop()
state := restore(task.State)
actions := discoverActions(state)
for _, action := range actions {
restore(state)
result := execute(action)
newState := observe(result)
addTransition(state, action, newState)
if isNewState(newState) {
queue.Push(newState)
}
}
}
7. Do Not Generate Every User Flow During Crawling
This was an important design decision.
The crawler should first discover the interaction graph.
For example:
A → B
A → C
B → D
C → D
D → E
Then a separate layer can generate paths:
A → B → D → E
A → C → D → E
Then an AI/flow-analysis layer can label them:
"User logs in and changes account settings"
"User searches for a product and checks out"
This keeps crawling and workflow reasoning separate.
8. Screen Mapper
The mapper should not depend exclusively on screenshots.
Three representations should be captured together:
DOM representation
button
text: "Continue"
role: button
boundingBox: ...
selector: ...
Accessibility representation
button "Continue"
textbox "Email"
link "Forgot password?"
Visual representation
screenshot.png
These should be combined into a canonical screen/interaction model.
Example:
{
"screen_id": "screen_123",
"url": "/checkout",
"elements": [
{
"id": "element_1",
"type": "button",
"text": "Continue",
"bbox": [720, 540, 120, 48],
"selector": "...",
"role": "button"
}
]
}
9. The Screen Mapper Should Become a Surface Observer
Once desktop automation was introduced, ScreenMapper was considered too web-specific.
The broader abstraction is:
type SurfaceObserver interface {
Observe(ctx context.Context) (*Observation, error)
}
An observation may contain:
type Observation struct {
SurfaceID SurfaceID
Screenshot ArtifactRef
Tree *InteractionTree
URL string
Window WindowInfo
Clipboard *ClipboardState
FocusedElement ElementID
Metadata map[string]any
}
For web:
DOM
Accessibility Tree
Screenshot
URL
Browser state
For desktop:
Accessibility Tree
Window hierarchy
Screenshot
Focused window
Focused control
Clipboard
10. Web + Desktop Unified Through Surface
The biggest architectural evolution was removing Browser as the fundamental abstraction.
Instead, introduce:
type SurfaceType string
const (
SurfaceWeb SurfaceType = "web"
SurfaceDesktop SurfaceType = "desktop"
SurfaceMobile SurfaceType = "mobile"
SurfaceTerminal SurfaceType = "terminal"
)
type Surface struct {
ID SurfaceID
Type SurfaceType
}
The kernel operates on surfaces rather than browsers.
Examples:
Web surface
Desktop surface
Mobile surface
Terminal surface
This allows future environments without redesigning the core domain.
11. Canonical Interaction Tree
Different environments should be normalized into a common interaction model.
Interaction Tree
│
┌───────────────────┼───────────────────┐
│ │ │
Web Desktop Mobile
│ │ │
DOM UIA/AX UI tree
│ │ │
└───────────────────┼───────────────────┘
↓
Canonical Elements
Core element model:
type Element struct {
ID ElementID
Role Role
Name string
Description string
Bounds Rect
Enabled bool
Visible bool
Focused bool
Capabilities []Capability
Parent ElementID
Children []ElementID
NativeRef any
}
The same conceptual element:
HTML button "Login"
and:
Windows UI button "Login"
should become:
Element {
Role: Button
Name: "Login"
Capabilities: [Click]
}
12. Action Model
Actions should also be environment-independent.
Instead of having:
ClickSelector()
MouseClick()
DOMClick()
as domain concepts, use semantic actions:
type Action struct {
ID ActionID
Target ElementID
Kind ActionKind
Input *ActionInput
Risk RiskLevel
}
Potential actions:
Click
DoubleClick
Type
Clear
Select
Check
Uncheck
Focus
Scroll
Drag
Drop
KeyPress
Hotkey
Submit
Wait
Launch
Close
Move
Resize
The runtime adapter translates the semantic action into the platform-specific operation.
13. Element Capabilities
Elements expose capabilities.
type Capability string
const (
CapClick Capability = "click"
CapType Capability = "type"
CapClear Capability = "clear"
CapFocus Capability = "focus"
CapSelect Capability = "select"
CapDrag Capability = "drag"
CapScroll Capability = "scroll"
CapKeyPress Capability = "keypress"
)
Examples:
Button:
click
focus
Textbox:
focus
type
clear
Slider:
focus
set_value
increment
decrement
Window:
focus
move
resize
minimize
maximize
close
This makes action discovery generic across web and desktop.
14. Desktop Runtime
Desktop automation requires a runtime abstraction.
Conceptually:
type DesktopRuntime interface {
Launch(ctx context.Context, app Application) error
Windows(ctx context.Context) ([]Window, error)
Observe(ctx context.Context) (*Observation, error)
Execute(ctx context.Context, action Action) error
Focus(ctx context.Context, window WindowID) error
Capture(ctx context.Context) (ArtifactRef, error)
Close(ctx context.Context, window WindowID) error
}
Platform-specific implementations can include:
Windows
├── UI Automation
└── Win32
macOS
├── Accessibility
└── Core Graphics
Linux
├── AT-SPI
├── X11
└── Wayland
The kernel must not depend directly on these APIs.
15. Desktop Mapping
Desktop applications don't necessarily have a DOM.
Therefore the observation pipeline becomes:
Native Accessibility Tree
+
Window Hierarchy
+
Screenshot
+
Input/Runtime Metadata
↓
Observation
↓
Screen Mapper
↓
Canonical Interaction Tree
For difficult applications:
Screenshot
↓
Vision Model
↓
Element Detection
↓
OCR
↓
Element Fusion
16. Mapper Plugin Model
Mapping should support multiple providers:
DOM Mapper
AX Mapper
UIA Mapper
AT-SPI Mapper
Vision Mapper
OCR Mapper
They can be combined:
Screenshot
│
┌──────────┴──────────┐
▼ ▼
OCR/Vision Accessibility
│ │
└──────────┬──────────┘
↓
Element Fusion
↓
Canonical UI Tree
This is especially important for:
canvas-based interfaces
Electron applications
legacy applications
custom controls
remote desktop applications
applications with poor accessibility support
17. Applications and Windows
Desktop automation introduces application boundaries.
A single user flow may be:
Chrome
↓
download file
↓
Finder / Explorer
↓
open file
↓
Excel
↓
edit cell
↓
save
Therefore applications and windows should become domain concepts.
type Application struct {
ID ApplicationID
Name string
Platform string
PID int
}
type Window struct {
ID WindowID
Application ApplicationID
Title string
Bounds Rect
}
The graph can therefore represent:
Chrome
│
│ download
↓
Filesystem
│
│ open
↓
Excel
This is one of the main reasons the architecture must not be web-specific.
18. Runtime State
Web and desktop have different state dimensions.
Web:
URL
Cookies
LocalStorage
SessionStorage
DOM
Authentication
Desktop:
Processes
Applications
Windows
Focused Window
Focused Element
Application State
Filesystem
Clipboard
OS dialogs
A generalized model can contain:
type RuntimeState struct {
Surface Surface
Applications []Application
Windows []Window
FocusedWindow WindowID
FocusedElement ElementID
BrowserState *BrowserState
OSState *OSState
}
However, not all environmental state should be included in the fingerprint.
The system needs to identify relevant state.
The entire filesystem, for example, should not cause every screen to become a unique state.
19. Resources
Desktop automation creates effects beyond the UI.
Important resources include:
File
Process
Window
Clipboard
Browser Session
External Service
Eventually:
Resource
├── File
├── Process
├── Window
├── Clipboard
├── BrowserSession
└── ExternalService
An action can then have:
Preconditions
UI Effects
Resource Effects
Example:
Click "Export"
↓
UI effect:
Download dialog appears
Resource effect:
File created
This makes workflows much more semantically useful.
20. Forms and Input Exploration
A crawler cannot generate arbitrary combinations of all possible form values.
Instead, input values should be generated using semantic classes.
Example:
email:
valid
invalid
empty
password:
valid
invalid
empty
search:
common query
nonsense query
empty
quantity:
1
0
maximum-ish
Later this can become a plugin:
input.generator
21. Exploration Budgets
Exploration must be bounded.
Example:
max_depth: 15
max_states: 10000
max_actions_per_screen: 50
max_time_per_site: 30m
max_retries: 2
max_form_variations: 5
Actions should have priorities.
Example:
login button → high
signup → high
primary CTA → high
navigation link → high
footer link → medium
social icon → low
external website → skip
mailto → skip
The scheduler is responsible for selecting what to explore next.
22. Exploration Scheduler
The crawler is effectively a scheduler over a graph.
Model:
type ExplorationTask struct {
StateID StateID
ActionID ActionID
Priority float64
Depth int
Risk RiskLevel
CreatedAt time.Time
}
Architecture:
Candidate Actions
↓
Scheduler
↓
┌─────┼─────┐
↓ ↓ ↓
Task A Task B Task C
Eventually AI can influence action priority, but it should not directly control the crawler.
23. AI / LLM Position
The LLM should not be the crawler.
Bad:
LLM
↓
look at screenshot
↓
decide everything
↓
click
↓
repeat
Better:
Browser/Desktop Runtime
↓
Deterministic Observation
↓
Candidate Actions
↓
LLM ranks/prioritizes
↓
Kernel validates
↓
Scheduler
↓
Runtime executes
↓
Deterministic State Capture
AI becomes a reasoning layer.
Possible AI plugins:
ScreenUnderstandingPlugin
ActionRankingPlugin
InputGenerationPlugin
StateSimilarityPlugin
FlowNamingPlugin
The kernel remains authoritative.
If the AI plugin is unavailable, the system should still be able to crawl deterministically.
24. Microkernel Decision
The architecture should use a Go microkernel.
The kernel owns:
Exploration
State
Graph
Scheduler
Policy
Workflow
Session
Plugin management
Events
The kernel should not know:
Playwright internals
Windows UI Automation APIs
macOS accessibility APIs
Python internals
Node internals
Postgres details
S3 details
NATS details
Those belong behind ports/adapters.
25. Ports & Adapters Decision
The architecture uses ports and adapters / hexagonal architecture.
Core ports:
Browser/Runtime
Observer
Mapper
Action Executor
Application
Filesystem
Process
Plugin
Graph Store
State Store
Artifact Store
Event Bus
Conceptually:
type Runtime interface {
Observe(ctx context.Context) (*Observation, error)
Execute(ctx context.Context, action Action) error
}
type SurfaceObserver interface {
Observe(ctx context.Context) (*Observation, error)
}
type ScreenMapper interface {
Map(ctx context.Context, observation Observation) (*ScreenMap, error)
}
type ActionDiscoverer interface {
Discover(ctx context.Context, screen Screen) ([]Action, error)
}
The domain/application layers depend on these ports, not implementations.
26. Web Runtime
The web runtime can initially use Playwright.
Web Runtime
└── Playwright
├── Chromium
├── Browser Context
├── Page
├── DOM
└── Accessibility Tree
The kernel only sees:
Runtime
Observation
Element
Action
State
Transition
27. Plugin Architecture
Python and Node plugins should run out-of-process.
Do not embed arbitrary Python/Node runtimes directly inside the Go kernel.
Reasons:
dependency isolation
independent runtime versions
crash isolation
memory isolation
GPU requirements
language-specific ecosystems
independent scaling
Architecture:
Go Kernel
│
Plugin Runtime
│
┌──┼───────────────┐
↓ ↓ ↓
Python Node Python
Vision LLM Mapper
28. Plugin Protocol
The initial recommendation was gRPC + Protocol Buffers.
Conceptually:
service Plugin {
rpc Describe(DescribeRequest)
returns (PluginDescriptor);
rpc Execute(ExecuteRequest)
returns (ExecuteResponse);
rpc Health(HealthRequest)
returns (HealthResponse);
}
Plugins advertise capabilities.
Example:
{
"name": "vision-mapper",
"runtime": "python",
"version": "1.2.0",
"capabilities": [
"screen.mapping",
"element.detection"
]
}
The kernel should not care whether the implementation is:
Python
Node
Go
Rust
GPU worker
Remote service
It only consumes capabilities.
29. Plugin Capability Model
Plugins should be organized around capabilities, not languages.
Examples:
browser
screen.mapper
action.discoverer
action.ranker
input.generator
state.similarity
flow.analyzer
vision
ocr
llm
reporter
exporter
Example manifest:
name: computer-vision-mapper
version: 1.0.0
runtime: python
capabilities:
- screen.mapper
- element.detector
protocol:
type: grpc
version: v1
resources:
gpu: optional
30. Plugin Authority
Plugins should not directly mutate kernel state.
Avoid:
Plugin
├── directly changes database
├── directly manipulates graph
└── directly controls crawler
Prefer:
Kernel
↓ request
Plugin
↓ result / observation
Kernel
↓ validates and decides
State / Graph
For example:
Screen captured
↓
Vision plugin
↓
"Detected button at x=420,y=320"
↓
Kernel validates
↓
Action added
Plugins provide observations and recommendations.
The kernel owns authority.
31. Event-Driven Architecture
Events should be used extensively, but not for every internal operation.
Domain events include:
ScreenDiscovered
ActionDiscovered
ActionStarted
ActionCompleted
ActionFailed
StateDiscovered
TransitionDiscovered
CoverageUpdated
FlowDiscovered
Operational events include:
PluginStarted
PluginFailed
BrowserCrashed
WorkerTimeout
RetryScheduled
The distinction is useful:
Domain events
Describe meaningful changes to the automation model.
Operational events
Describe infrastructure/runtime behavior.
32. Synchronous Exploration + Asynchronous Events
A critical architectural decision:
The core exploration loop should remain synchronous.
Do not build the primary crawler as a Kafka pipeline like:
ActionDiscovered
→ Kafka
→ Worker
→ Kafka
→ ActionExecuted
→ Kafka
→ StateCaptured
→ Kafka
That introduces unnecessary latency and complexity.
Instead:
Kernel
↓
Explore
↓
Execute
↓
Observe
↓
Resolve State
↓
Update Graph
Then publish events for side effects:
ActionCompleted
↓
┌──┼──────────────┐
↓ ↓ ↓
Graph Metrics UI
Writer Coverage
33. Event Bus
The first version can use an in-process event bus.
Later it can be replaced by NATS.
The architecture should therefore have:
EventBus interface
with implementations:
InMemoryEventBus
NATSAdapter
This lets the system become distributed without changing the domain model.
34. NATS Decision
If distributed execution becomes necessary, NATS was preferred over immediately adopting Kafka.
Reasons:
lightweight pub/sub
request/reply
work queues
simple operational model
suitable for worker coordination
Kafka can still become useful if the system eventually requires very large-scale event-stream processing.
But it should not be a prerequisite for the initial architecture.
35. Storage Decision
Start with PostgreSQL.
Store:
sites
runs
surfaces
applications
windows
screens
elements
states
actions
transitions
flows
plugins
events
Use object storage for large artifacts:
screenshots
DOM snapshots
accessibility trees
videos
HAR files
browser traces
plugin artifacts
Examples:
S3-compatible storage
Filesystem
Redis should only be introduced if actually required for:
distributed scheduling
locks
caching
queues
36. Graph Storage Decision
Do not start with Neo4j purely because the product contains a graph.
Keep the graph domain independent of storage.
Start with PostgreSQL.
Conceptual domain:
Screen
│
├── contains → Element
│
└── exposes → Action
│
└── leads to → Screen
If graph-specific queries later become a bottleneck, a graph database can be added as an adapter/projection.
37. Domain Model
The final domain should look approximately like:
Site
│
└── Run
│
├── Surface
│ │
│ ├── Application
│ └── Window
│
├── State
│ │
│ └── Screen
│ │
│ └── Element
│
├── Action
│
├── Transition
│
└── Flow
The broader interaction model is:
Surface
Application
Window
Element
Action
State
Transition
Flow
Resource
38. Risk and Policy
Desktop automation increases the danger of destructive actions.
Actions should be classified:
SAFE
navigation
open menu
tabs
search
CAUTION
add to cart
submit form
send message
DANGEROUS
delete account
delete data
purchase
send email
transfer money
Initial policy:
SAFE → automatic
CAUTION → configurable
DANGEROUS → denied by default
Policy belongs in the Go kernel.
It must not be delegated to plugins.
39. Desktop Security
Desktop automation may interact with:
filesystem
credentials
clipboard
processes
network
system settings
email
financial applications
Serious deployments should therefore support isolation such as:
VM
sandbox
dedicated automation account
allowlisted applications
filesystem restrictions
network restrictions
The policy engine should be evaluated before executing an action.
Action
↓
Policy Engine
├── allowed
├── requires approval
└── denied
40. Session / Authentication Model
Web crawling requires multiple session profiles.
Examples:
anonymous
user_A
admin
premium_user
Authenticated and anonymous exploration should not accidentally share browser state.
For desktop, a similar concept applies:
OS user
Application profile
Browser profile
Credential context
The session becomes a first-class concept.
41. Final Go Project Structure
A proposed structure:
automation/
│
├── cmd/
│ └── automation/
│
├── kernel/
│ ├── domain/
│ │ ├── surface/
│ │ ├── application/
│ │ ├── window/
│ │ ├── element/
│ │ ├── action/
│ │ ├── state/
│ │ ├── transition/
│ │ ├── flow/
│ │ ├── resource/
│ │ └── run/
│ │
│ ├── application/
│ │ ├── explorer/
│ │ ├── scheduler/
│ │ ├── mapper/
│ │ ├── coverage/
│ │ ├── policy/
│ │ └── workflow/
│ │
│ ├── ports/
│ │ ├── runtime.go
│ │ ├── observer.go
│ │ ├── mapper.go
│ │ ├── executor.go
│ │ ├── application.go
│ │ ├── filesystem.go
│ │ ├── process.go
│ │ ├── plugin.go
│ │ ├── graph.go
│ │ ├── state.go
│ │ ├── artifacts.go
│ │ └── events.go
│ │
│ └── events/
│ ├── events.go
│ └── bus.go
│
├── adapters/
│ ├── web/
│ │ └── playwright/
│ │
│ ├── desktop/
│ │ ├── windows/
│ │ │ ├── uia/
│ │ │ └── win32/
│ │ │
│ │ ├── macos/
│ │ │ └── accessibility/
│ │ │
│ │ └── linux/
│ │ ├── atspi/
│ │ ├── x11/
│ │ └── wayland/
│ │
│ ├── persistence/
│ │ └── postgres/
│ │
│ ├── artifacts/
│ │ ├── filesystem/
│ │ └── s3/
│ │
│ ├── messaging/
│ │ ├── memory/
│ │ └── nats/
│ │
│ └── plugins/
│ └── grpc/
│
├── proto/
│ └── plugin/
│
└── plugins/
├── python/
└── node/
42. Final Runtime Architecture
┌─────────────────────────────────────────────────────────────┐
│ GO KERNEL │
│ │
│ Domain │
│ ├── Surface │
│ ├── Application │
│ ├── Window │
│ ├── Element │
│ ├── Action │
│ ├── State │
│ ├── Transition │
│ ├── Flow │
│ └── Resource │
│ │
│ Engines │
│ ├── Explorer │
│ ├── Scheduler │
│ ├── State Resolver │
│ ├── Action Discovery │
│ ├── Policy │
│ ├── Coverage │
│ └── Workflow │
│ │
│ Ports │
│ ├── Runtime │
│ ├── Observer │
│ ├── Mapper │
│ ├── Executor │
│ ├── Resource │
│ ├── Plugin │
│ └── Storage │
└──────────────────────────┬──────────────────────────────────┘
│
Event Bus
│
┌───────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Web Runtime │ │Desktop Runtime│ │ Future │
│ │ │ │ │ Runtimes │
│ Playwright │ │ Windows UIA │ │ Mobile │
│ Chromium │ │ macOS AX │ │ Terminal │
└─────────────┘ │ Linux AT-SPI │ │ Remote │
└──────────────┘ └──────────────┘
┌─────────────────────────────────────────────┐
│ Plugin Processes │
│ │
│ Python │ Node │ Go │ GPU │ Remote │
│ │
│ Vision │ OCR │ LLM │ Mapper │ Analyzer │
└─────────────────────────────────────────────┘
43. Final Exploration Loop
The fundamental runtime loop is:
┌─────────────────────┐
│ Observe │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Understand │
│ DOM / AX / Vision │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Discover Actions │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Policy │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Scheduler │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Execute │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Observe │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Resolve State │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Update Graph │
└──────────┬──────────┘
│
└──────→ next task
44. Final Architectural Principles
These are the decisions to treat as architectural invariants.
1. Go owns the kernel
Go owns:
orchestration
domain state
graph
scheduler
policy
events
sessions
workflow
2. Browser is not the fundamental abstraction
Use:
Surface
rather than:
Browser
This enables desktop, mobile, terminal, and future runtimes.
3. The graph is the core product
The crawler discovers:
States
Actions
Transitions
The workflow engine derives:
User Flows
4. Use Ports & Adapters
The kernel should never directly depend on:
Playwright
UIA
macOS Accessibility
AT-SPI
Postgres
S3
NATS
Python
Node
5. Plugins are out-of-process
Use:
gRPC + protobuf
for the initial plugin protocol.
6. Plugins are capability-based
Examples:
screen.mapper
vision
ocr
action.ranker
input.generator
state.similarity
flow.analyzer
7. Plugins are advisory
Plugins return observations/recommendations.
The kernel remains authoritative.
8. Exploration stays synchronous
The critical loop:
observe → decide → execute → observe
should remain low-latency and deterministic.
9. Events are asynchronous side effects
Use events for:
metrics
coverage
UI updates
logging
audit
projections
external integrations
10. Start with an in-process event bus
Later provide:
NATS adapter
without changing the domain.
11. Start with PostgreSQL
Do not introduce a graph database, Kafka, Redis, or microservices prematurely.
12. Artifacts belong in object storage
Screenshots, videos, DOM snapshots, traces, etc. should not bloat the relational database.
13. State identity must be explicit
A URL is not a state.
14. AI should rank and reason, not own the crawler
The deterministic automation engine should remain operational even when AI is unavailable.
15. Policy belongs in the kernel
Especially for desktop automation.
16. Desktop and web should share the same domain model
The goal is:
Web
Desktop
Mobile
Terminal
Remote Desktop
...
↓
same
↓
Surface → Observation → Element → Action → State → Transition
45. Final One-Line Architecture Decision
The final architecture can be summarized as:
A Go modular microkernel using Ports & Adapters for runtime abstraction, an event-driven side-effect architecture, a synchronous graph-based exploration engine, and isolated Python/Node capability plugins communicating through gRPC—where Web, Desktop, and future automation environments are runtime adapters over a common Surface/Interaction/State model.
And the most important conceptual shift is:
Not:
Web Crawler + Desktop Crawler
But:
┌──────────────────────────┐
│ Automation Engine │
└────────────┬─────────────┘
│
Interaction Graph
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Web Desktop Mobile
Runtime Runtime Runtime
This gives you a foundation for eventually building something closer to a general-purpose autonomous application exploration and automation platform, rather than a crawler tied to one UI technology.5 views