Report a problem your own scripts find
A signal is one record per condition. Something on your side finds a problem, POSTs it to DedNets with a name you choose, and the Console notifies you. When the same script finds the same problem again five minutes later, it POSTs again and nobody is notified: the existing signal just counts up. When the condition clears, the signal closes and you get one all-clear. (Since v0.9.0.)
That is the whole idea, and it is what separates a signal from its two neighbours:
- A notification is one message per event. It has no memory, so a check that runs every 15 minutes sends you 96 messages a day.
- A watchdog waits for a ping that should arrive and alerts on silence.
- A signal waits for a report that should not arrive, and alerts once per episode of it.
The case this was built for: a cron job on a server that scans the SSH auth log for brute-force attempts. It runs every 15 minutes. You want to be told the first time it finds something, you want to know how many attempts it saw and from where, and you do not want to be told again every 15 minutes until you fix it.
1. Create a reporting URL
Section titled “1. Create a reporting URL”A reporting URL is what your script posts to. Possession of it is the whole credential, exactly like a watchdog’s ping URL, so it goes on the machine that does the reporting.
In the Console, open Reporting URLs (the button of that name on the Signals page) and add one named after the machine or the job that will use it. Or from the CLI:
dednetsctl signals source add web1Either way you get a URL like
https://console.dednets.com/api/v0/sig/sg<32 hex characters>.
It is write-only: it can report a condition and nothing else. It cannot list, resolve or delete anything. That is deliberate, and it matters most in exactly the case this feature exists for: if the machine doing the reporting is itself the thing under attack, whoever takes it over still cannot use that URL to silence the alarm it just tripped.
Keep it out of public repositories and logs. There is no rotate: revoke it and add another. Revoking keeps every signal it already reported.
A script or an AI agent can mint one for itself with an API token carrying the
Signals reporting URLs (signals:sources) scope (since v0.14.7); before
that it took your login. That is a separate scope from Signals on purpose:
Signals lets something report a condition, while this one lets it hand out
new URLs that report conditions. See
API tokens and scopes.
2. Report a condition
Section titled “2. Report a condition”Append the key to the URL and POST:
curl -sS -X POST -H "X-Severity: error" -H "X-Title: SSH brute force on web1" \ -d '{"ip":"203.0.113.9","attempts":412}' \ https://console.dednets.com/api/v0/sig/sg<32hex>/sshd-bruteforce-web1That is the whole integration. Run it from cron, from a systemd timer, from a CI job, from anything that can make an HTTP request.
You can also send one JSON object instead of headers plus a body:
curl -sS -X POST -H "Content-Type: application/json" \ -d '{"severity":"error","title":"SSH brute force on web1", "payload":{"ip":"203.0.113.9","attempts":412}}' \ https://console.dednets.com/api/v0/sig/sg<32hex>/sshd-bruteforce-web1What each part does:
- severity is
info,warning(the default),errororcritical. It sets how loudly the notification arrives: priority 2, 3, 4 and 5 on the same scale your other notifications use. - title is the one-line summary that becomes the notification’s title.
- detail is longer text, and becomes the notification’s body.
- payload is a JSON object of whatever you want to keep: the Console shows it as name/value rows on the signal’s page and includes it in the alert that wakes you. It must be an object, not a list, and at most 8 KB.
- group is an optional label that buckets related signals in the Console.
- ttl is covered in step 4.
You only have to send these once. A field you leave out of a later report
leaves what the Console already has alone, so the short form - a payload and
nothing else - is a perfectly good repeat: it will not wipe your title, ungroup
your incident, downgrade an error to warning, or shorten your TTL. Send a
field again only when you want to change it. (Since v0.9.1. Before that,
leaving out severity or ttl reset them to the defaults.)
3. Choose a good key
Section titled “3. Choose a good key”This is the one thing worth getting right, and the one thing that is easy to get wrong in a way that undoes the whole feature.
The key names the condition, not the event, so it must be identical on every report of the same problem. Keep anything that varies per occurrence out of it and put it in the payload instead.
| Do | Don’t |
|---|---|
sshd-bruteforce-web1 |
sshd-bruteforce-203.0.113.9-1699999999 |
disk-full-vm1 |
disk-full-vm1-93-percent |
backup-verify-failed |
backup-verify-failed-run-4471 |
The failure mode is specific: a key with a timestamp, a count or an attacker’s IP in it is a brand new signal every single time, and a brand new notification with it. You will have rebuilt the firehose you came here to escape, and the Console will fill up with hundreds of one-occurrence signals.
If two machines report the same kind of problem and you want to tell them
apart, put the machine in the key (disk-full-web1, disk-full-vm1). Signals on
one account share one keyspace, so without that they fold into a single record.
Keys use letters, digits, - and _, up to 64 characters.
4. Closing it
Section titled “4. Closing it”A signal closes in one of two ways, and you can use both.
It closes itself. Every signal has a TTL: if nothing reports it for that
long, the Console resolves it and sends one all-clear. The default is one hour,
which is four missed cycles for a job that runs every 15 minutes. Set your own
with -H "X-TTL: 7200" or "ttl": 7200 in the JSON body (60 seconds to 30
days).
Pick a TTL comfortably longer than how often your script runs. A TTL shorter than the reporting interval makes a signal flap closed and open again, which notifies you on every cycle.
You set the TTL on the report that OPENS the signal; later reports that do not mention it keep the window you chose.
Or it never closes by itself. Some problems do not go away when the script
stops reporting them: a new bad block found by SMART is still there tomorrow.
Report those with -H "X-Auto-Resolve: false" (or "auto_resolve": false in
JSON, or dednetsctl signals send <key> -no-auto-resolve) and the signal stays
open, however long nothing reports it, until you resolve it yourself.
(Since v0.16.1.)
curl -sS -X POST -H "X-Severity: critical" -H "X-Auto-Resolve: false" \ -H "X-Title: New bad block on sda" -d '{"reallocated":3}' \ https://console.dednets.com/api/v0/sig/sg<32 hex>/smart-bad-block-sdaLater reports that do not mention it keep the setting. To change it on an
existing signal, use Edit in the Console (the Auto-resolve switch), or
dednetsctl signals update <key> -auto-resolve=false (=true turns the TTL
back on). A signal like this is also kept past the 30-day clean-up for as long
as it is open.
Or you close it explicitly, which is how you say “this cleared” without waiting out the TTL. From the Console’s Resolve button, or:
dednetsctl signals resolve sshd-bruteforce-web1Resolving is idempotent and non-destructive. Resolving an already-resolved signal changes nothing and notifies nobody, so a script can call it unconditionally at the end of every run:
if scan_auth_log; then dednetsctl signals resolve sshd-bruteforce-web1 # all clear, every cycleelse dednetsctl signals send sshd-bruteforce-web1 -severity errorfiAnd if the condition comes back, reporting the same key reopens the same
record rather than creating a new one. Its history is kept, and it starts a
new episode: the count goes back to 1 and you are notified again. The
Console’s model of sshd-bruteforce-web1 firing again is “this is back”, not
“here is an unrelated problem with the same name”.
5. See it in the Console
Section titled “5. See it in the Console”The Signals page lists one row per condition, showing its severity, whether it is still open, and how many times it has been reported in the current episode. It defaults to Open, because the question the list answers is “what is wrong right now”; switch to All to see what has already resolved.
Click a signal for its full payload and its history. The history records state changes only - opened, reopened, resolved, auto-resolved - not occurrences. A condition reported every 15 minutes for a week has a handful of rows there, not seven hundred.
Alerts land in your signals notification topic, so they fan out exactly
like everything else: the Console feed, your phone, and Telegram if you have
linked it. To keep low-severity ones off your phone, set that topic’s minimum
priority in Notifications.
Note the default is to send everything, so info signals do reach a linked
Telegram chat until you change it.
Add a description (since v0.16.3)
Section titled “Add a description (since v0.16.3)”Give a signal a description: your own note on what the condition means and what to do when it fires (“check the fail2ban jail, then rotate the keys”). Open the signal, click Edit, and fill in Description; up to 1000 characters, line breaks allowed. The list shows its first line, the signal’s page shows it whole.
The description belongs to the condition, not to one report: your script never sends it, and no report changes or clears it, so it stays through repeats, resolve and reopen. Clear it by emptying the field.
One thing worth knowing: raising the severity of a signal that is already open does not notify you again. If it did, a script whose severity flapped would page you every cycle. When a severe case deserves its own alert, report it under its own key.
6. Quiet windows: expected alerts that should not page you
Section titled “6. Quiet windows: expected alerts that should not page you”Some signals fire for a known reason at a known time. An IOPS signal that trips every night during the full backup is real, but it is not news. Give that signal a quiet window and it stops paging you for it. (Since v0.16.0.)
Inside a quiet window:
- a report still opens the signal and is recorded in its history and in the Alerts history (as Opened, followed by Deferred);
- no notification is sent, and the signal stays out of Needs you now;
- if the signal is still open when the window ends, you get the notification then, titled “Signal still open after quiet window”;
- if it resolves inside the window, by itself or because your script resolved it, you are notified of neither the open nor the all-clear.
So the backup that pushes IOPS up for an hour and then calms down pages nobody, while a disk that is still struggling an hour after the backup ended still reaches you.
A signal can have up to 4 windows. Each is a start and an end time, an
optional set of weekdays, and a time zone. An end earlier than the start
crosses midnight, and belongs to the day it starts on: 23:30-02:00 on
Friday covers Friday night into Saturday morning. With no weekdays it applies
every day.
In the Console, open the signal, click Edit, and add a window under Quiet windows. The signal’s page then shows its windows, and a Quiet until 02:00 or Notification deferred badge while one is in effect; in the list, a moon marks a signal that has windows.
From the CLI, pass each window as "HH:MM-HH:MM [days] [zone]":
dednetsctl signals update disk-iops-web1 \ -quiet "23:30-02:00 mon-fri America/Sao_Paulo" \ -quiet "01:00-05:00 sat,sun America/Sao_Paulo"dednetsctl signals update disk-iops-web1 -quiet-clear # remove them allDays are mon to sun, as a range (mon-fri) or a list (sat,sun). Leave the
zone out and your computer’s time zone is used. -quiet replaces every window
the signal had, so pass all of them each time.
From the API, PATCH /api/v1/signals/{id} with
"quiet_windows": [{"start":"23:30","end":"02:00","days":["mon","fri"],"tz":"America/Sao_Paulo"}]
([] clears them). A report made inside a window answers
"notified": false, "deferred": true. From an AI agent, ask it to set quiet
windows on the signal; it uses signal_update.
A signal must have been reported at least once before you can give it a window: a signal only exists after its first report. For the backup case that is usually the night it first paged you.
7. From the CLI
Section titled “7. From the CLI”dednetsctl signals list # open signalsdednetsctl signals list -state all # including resolveddednetsctl signals show sshd-bruteforce-web1 # payload and historydednetsctl signals send disk-full-web1 -severity warning \ -payload '{"free_gb":2.1}'dednetsctl signals resolve disk-full-web1dednetsctl signals update disk-full-web1 \ -description "Free space on /var, then check the log rotation" # since v0.16.3dednetsctl signals update disk-full-web1 -description "" # clear itsignals show prints the description whole; signals list adds a clipped
DESCRIPTION column when any listed signal has one.
signals send prints the one thing curl cannot tell you: whether your report
opened a signal or folded into one that was already open. That is what
makes it the right tool while you are wiring an integration up, even though
curl is what runs in production.
The reads need a token with the read or signals scope; reporting, resolving
and updating need signals. Deleting a signal and managing reporting URLs need
your own login rather than a token - see API tokens and
scopes.
8. From an AI agent
Section titled “8. From an AI agent”An agent connected to DedNets can list your signals, read one in full, close one, report one, and set or clear a signal’s description (since v0.16.3). See Let an AI agent run DedNets.
Worth knowing before you ask an agent to test the wiring: signal_send posts a
real occurrence and can reach your phone. Ask it to report to a throwaway
key you can delete afterwards.
Limits
Section titled “Limits”- Reporting URLs are capped per account (50 by default). Delete one you no longer use to make room; the Console answers a create past the cap with a quota error. (Since v0.14.11.)
- Signals are per account, not per reporting URL. Two machines reporting
disk-fullfold into one signal; put the machine in the key when that is not what you want. - There is no bulk resolve yet. Closing forty signals after one fix is forty clicks or forty CLI calls.
- Quiet windows recur; there is no one-off snooze (“mute until 18:00
today”). Besides quiet windows, the only volume knob is the
signalstopic’s minimum priority. - Signals are kept for 30 days after their last activity, then removed automatically, whether they ended up resolved or not. “Last activity” means the most recent report, or the moment the signal resolved. A condition that keeps being reported is never old, however long ago it first fired, so an incident that has run for months is not at risk. A signal set to never auto-resolve is kept for as long as it is open. (Since v0.9.1. Before that, resolved signals were kept for 30 days and open ones were kept forever.)