Yardbolt
Yardbolt is a query fabric for security telemetry. Query gateway would be another way of describing it. The name comes from yards of fabric and bolts of cloth. The idea is to manage connectivity to different defense tools through a single system and allow you to write one generic domain language query that runs against whichever defense tools you have. It wraps stix-shifter to translate a single STIX pattern into specific vendor queries, runs searches as async jobs, and presents results through a Vue UI. This is kind of a weird use of stix-shifter since it's intended to be CTI / indicator focused, but the library handles all the connectivity with all these different tools so you can use it to write native queries as well. My thinking was I could use this to check logs after attack simulations more easily than writing a big pile of integrations. BAS control validation is typically more blind and this gives you something a bit closer to a realistic audit but not as time-consuming as manual incident response. There's some Sigma rule integration as well. This was my first python project so I was learning the language by building Yardbolt. I'd heard a little bit about FastAPI (very very new at the time) so I bought into SQLModel as well. Today, I'd probably just use SQLAlchemy directly.
Fig. 1: Primary Interface
⤢ ZOOMYardbolt solves a problem I kept seeing with attack simulations where events were (hopefully) in the SIEM, but almost every tool has a different query language. stix-shifter does translation and SIGMA sort-of does too. Yardbolt uses those while adding on accounts and tool connections, saved queries, async execution, and result views for the queries and a UI. I picked it up as my way into Python, but I put it on the shelf for a bit when I thought SIEMs were starting to standardize around common schemas like ECS. A couple years later and this correlation process is still a bit less solved than I'd thought it would be.
Tech Detail
Fig. 2: Functional Overview
⤢ ZOOM
| Backend | FastAPI + SQLModel (async) |
| Query layer | stix-shifter |
| Jobs | arq workers on Redis |
| Store | PostgreSQL + Alembic |
| Frontend | Vue 3 / Vite / Pinia / PrimeVue |
| Detections | pySigma -> backend rules |
| Year | 2019-21 |
| Visibility | Proprietary preview |
Notes
Queries can take a long time on prod systems and I didn't want their slowness to be a problem. I made it so the query gets written to the database, and then it hands the work to an arq worker on Redis, and returns a job id. The worker runs the search, normalizes results, and updates the record while the frontend polls for it with a backoff that starts at a few seconds and stretches out. This avoids hammering the server while SIEM or log aggregator takes its time.
This was my first Python project and I started it on async FastAPI when the framework was about a year old and async SQLAlchemy juuust starting. I hit a lot of walls with early FastAPI, and SQLModel had a ton of problems, but ORMs all have a lot of tradeoffs so I kept complex queries as separate manually-constructed prepared statements.
The Vue client is generated from the backend's OpenAPI spec, so every endpoint and model goes through a typed client instead of rando axios calls. I used CodeMirror for query authoring and Chart.js for the dashboard.
Screens & Reference
Fig. 3: Dashboard
⤢ ZOOMI was working on splitting query results into incidents, alerts, blocks, detections, queried logs with a recent-activity feed of submitted queries and logged incidents.
Fig. 4: Query Builder
⤢ ZOOMYou can build queries against connected defense tool accounts, tagging them with ATT&CK techniques or other information that give you a customizable result view.
Fig. 5: Defense Tools
⤢ ZOOMStix shifter tool integrations discovered from the stix-shifter library modules installed at runtime.
Fig. 6: Result Views
⤢ ZOOMA drag-and-drop builder for saved result views which let you configure datatable columns and row-expansion fields mapped from a tool's result schema.