Skip to content

Report a problem your own scripts find

A signal is one record per condition. Something on your side finds a problem, POSTs it to DedNets with a name you choose, and the Console notifies you. When the same script finds the same problem again five minutes later, it POSTs again and nobody is notified: the existing signal just counts up. When the condition clears, the signal closes and you get one all-clear. (Since v0.9.0.)

That is the whole idea, and it is what separates a signal from its two neighbours:

  • A notification is one message per event. It has no memory, so a check that runs every 15 minutes sends you 96 messages a day.
  • A watchdog waits for a ping that should arrive and alerts on silence.
  • A signal waits for a report that should not arrive, and alerts once per episode of it.

The case this was built for: a cron job on a server that scans the SSH auth log for brute-force attempts. It runs every 15 minutes. You want to be told the first time it finds something, you want to know how many attempts it saw and from where, and you do not want to be told again every 15 minutes until you fix it.

A reporting URL is what your script posts to. Possession of it is the whole credential, exactly like a watchdog’s ping URL, so it goes on the machine that does the reporting.

In the Console, open Reporting URLs (the button of that name on the Signals page) and add one named after the machine or the job that will use it. Or from the CLI:

Terminal window
dednetsctl signals source add web1

Either way you get a URL like https://console.dednets.com/api/v0/sig/sg<32 hex characters>.

It is write-only: it can report a condition and nothing else. It cannot list, resolve or delete anything. That is deliberate, and it matters most in exactly the case this feature exists for: if the machine doing the reporting is itself the thing under attack, whoever takes it over still cannot use that URL to silence the alarm it just tripped.

Keep it out of public repositories and logs. There is no rotate: revoke it and add another. Revoking keeps every signal it already reported.

A script or an AI agent can mint one for itself with an API token carrying the Signals reporting URLs (signals:sources) scope (since v0.14.7); before that it took your login. That is a separate scope from Signals on purpose: Signals lets something report a condition, while this one lets it hand out new URLs that report conditions. See API tokens and scopes.

Append the key to the URL and POST:

Terminal window
curl -sS -X POST -H "X-Severity: error" -H "X-Title: SSH brute force on web1" \
-d '{"ip":"203.0.113.9","attempts":412}' \
https://console.dednets.com/api/v0/sig/sg<32hex>/sshd-bruteforce-web1

That is the whole integration. Run it from cron, from a systemd timer, from a CI job, from anything that can make an HTTP request.

You can also send one JSON object instead of headers plus a body:

Terminal window
curl -sS -X POST -H "Content-Type: application/json" \
-d '{"severity":"error","title":"SSH brute force on web1",
"payload":{"ip":"203.0.113.9","attempts":412}}' \
https://console.dednets.com/api/v0/sig/sg<32hex>/sshd-bruteforce-web1

What each part does:

  • severity is info, warning (the default), error or critical. It sets how loudly the notification arrives: priority 2, 3, 4 and 5 on the same scale your other notifications use.
  • title is the one-line summary that becomes the notification’s title.
  • detail is longer text, and becomes the notification’s body.
  • payload is a JSON object of whatever you want to keep: the Console shows it as name/value rows on the signal’s page and includes it in the alert that wakes you. It must be an object, not a list, and at most 8 KB.
  • group is an optional label that buckets related signals in the Console.
  • ttl is covered in step 4.

You only have to send these once. A field you leave out of a later report leaves what the Console already has alone, so the short form - a payload and nothing else - is a perfectly good repeat: it will not wipe your title, ungroup your incident, downgrade an error to warning, or shorten your TTL. Send a field again only when you want to change it. (Since v0.9.1. Before that, leaving out severity or ttl reset them to the defaults.)

This is the one thing worth getting right, and the one thing that is easy to get wrong in a way that undoes the whole feature.

The key names the condition, not the event, so it must be identical on every report of the same problem. Keep anything that varies per occurrence out of it and put it in the payload instead.

Do Don’t
sshd-bruteforce-web1 sshd-bruteforce-203.0.113.9-1699999999
disk-full-vm1 disk-full-vm1-93-percent
backup-verify-failed backup-verify-failed-run-4471

The failure mode is specific: a key with a timestamp, a count or an attacker’s IP in it is a brand new signal every single time, and a brand new notification with it. You will have rebuilt the firehose you came here to escape, and the Console will fill up with hundreds of one-occurrence signals.

If two machines report the same kind of problem and you want to tell them apart, put the machine in the key (disk-full-web1, disk-full-vm1). Signals on one account share one keyspace, so without that they fold into a single record.

Keys use letters, digits, - and _, up to 64 characters.

A signal closes in one of two ways, and you can use both.

It closes itself. Every signal has a TTL: if nothing reports it for that long, the Console resolves it and sends one all-clear. The default is one hour, which is four missed cycles for a job that runs every 15 minutes. Set your own with -H "X-TTL: 7200" or "ttl": 7200 in the JSON body (60 seconds to 30 days).

Pick a TTL comfortably longer than how often your script runs. A TTL shorter than the reporting interval makes a signal flap closed and open again, which notifies you on every cycle.

You set the TTL on the report that OPENS the signal; later reports that do not mention it keep the window you chose.

Or it never closes by itself. Some problems do not go away when the script stops reporting them: a new bad block found by SMART is still there tomorrow. Report those with -H "X-Auto-Resolve: false" (or "auto_resolve": false in JSON, or dednetsctl signals send <key> -no-auto-resolve) and the signal stays open, however long nothing reports it, until you resolve it yourself. (Since v0.16.1.)

Terminal window
curl -sS -X POST -H "X-Severity: critical" -H "X-Auto-Resolve: false" \
-H "X-Title: New bad block on sda" -d '{"reallocated":3}' \
https://console.dednets.com/api/v0/sig/sg<32 hex>/smart-bad-block-sda

Later reports that do not mention it keep the setting. To change it on an existing signal, use Edit in the Console (the Auto-resolve switch), or dednetsctl signals update <key> -auto-resolve=false (=true turns the TTL back on). A signal like this is also kept past the 30-day clean-up for as long as it is open.

Or you close it explicitly, which is how you say “this cleared” without waiting out the TTL. From the Console’s Resolve button, or:

Terminal window
dednetsctl signals resolve sshd-bruteforce-web1

Resolving is idempotent and non-destructive. Resolving an already-resolved signal changes nothing and notifies nobody, so a script can call it unconditionally at the end of every run:

Terminal window
if scan_auth_log; then
dednetsctl signals resolve sshd-bruteforce-web1 # all clear, every cycle
else
dednetsctl signals send sshd-bruteforce-web1 -severity error
fi

And if the condition comes back, reporting the same key reopens the same record rather than creating a new one. Its history is kept, and it starts a new episode: the count goes back to 1 and you are notified again. The Console’s model of sshd-bruteforce-web1 firing again is “this is back”, not “here is an unrelated problem with the same name”.

The Signals page lists one row per condition, showing its severity, whether it is still open, and how many times it has been reported in the current episode. It defaults to Open, because the question the list answers is “what is wrong right now”; switch to All to see what has already resolved.

Click a signal for its full payload and its history. The history records state changes only - opened, reopened, resolved, auto-resolved - not occurrences. A condition reported every 15 minutes for a week has a handful of rows there, not seven hundred.

Alerts land in your signals notification topic, so they fan out exactly like everything else: the Console feed, your phone, and Telegram if you have linked it. To keep low-severity ones off your phone, set that topic’s minimum priority in Notifications. Note the default is to send everything, so info signals do reach a linked Telegram chat until you change it.

Give a signal a description: your own note on what the condition means and what to do when it fires (“check the fail2ban jail, then rotate the keys”). Open the signal, click Edit, and fill in Description; up to 1000 characters, line breaks allowed. The list shows its first line, the signal’s page shows it whole.

The description belongs to the condition, not to one report: your script never sends it, and no report changes or clears it, so it stays through repeats, resolve and reopen. Clear it by emptying the field.

One thing worth knowing: raising the severity of a signal that is already open does not notify you again. If it did, a script whose severity flapped would page you every cycle. When a severe case deserves its own alert, report it under its own key.

6. Quiet windows: expected alerts that should not page you

Section titled “6. Quiet windows: expected alerts that should not page you”

Some signals fire for a known reason at a known time. An IOPS signal that trips every night during the full backup is real, but it is not news. Give that signal a quiet window and it stops paging you for it. (Since v0.16.0.)

Inside a quiet window:

  • a report still opens the signal and is recorded in its history and in the Alerts history (as Opened, followed by Deferred);
  • no notification is sent, and the signal stays out of Needs you now;
  • if the signal is still open when the window ends, you get the notification then, titled “Signal still open after quiet window”;
  • if it resolves inside the window, by itself or because your script resolved it, you are notified of neither the open nor the all-clear.

So the backup that pushes IOPS up for an hour and then calms down pages nobody, while a disk that is still struggling an hour after the backup ended still reaches you.

A signal can have up to 4 windows. Each is a start and an end time, an optional set of weekdays, and a time zone. An end earlier than the start crosses midnight, and belongs to the day it starts on: 23:30-02:00 on Friday covers Friday night into Saturday morning. With no weekdays it applies every day.

In the Console, open the signal, click Edit, and add a window under Quiet windows. The signal’s page then shows its windows, and a Quiet until 02:00 or Notification deferred badge while one is in effect; in the list, a moon marks a signal that has windows.

From the CLI, pass each window as "HH:MM-HH:MM [days] [zone]":

Terminal window
dednetsctl signals update disk-iops-web1 \
-quiet "23:30-02:00 mon-fri America/Sao_Paulo" \
-quiet "01:00-05:00 sat,sun America/Sao_Paulo"
dednetsctl signals update disk-iops-web1 -quiet-clear # remove them all

Days are mon to sun, as a range (mon-fri) or a list (sat,sun). Leave the zone out and your computer’s time zone is used. -quiet replaces every window the signal had, so pass all of them each time.

From the API, PATCH /api/v1/signals/{id} with "quiet_windows": [{"start":"23:30","end":"02:00","days":["mon","fri"],"tz":"America/Sao_Paulo"}] ([] clears them). A report made inside a window answers "notified": false, "deferred": true. From an AI agent, ask it to set quiet windows on the signal; it uses signal_update.

A signal must have been reported at least once before you can give it a window: a signal only exists after its first report. For the backup case that is usually the night it first paged you.

Terminal window
dednetsctl signals list # open signals
dednetsctl signals list -state all # including resolved
dednetsctl signals show sshd-bruteforce-web1 # payload and history
dednetsctl signals send disk-full-web1 -severity warning \
-payload '{"free_gb":2.1}'
dednetsctl signals resolve disk-full-web1
dednetsctl signals update disk-full-web1 \
-description "Free space on /var, then check the log rotation" # since v0.16.3
dednetsctl signals update disk-full-web1 -description "" # clear it

signals show prints the description whole; signals list adds a clipped DESCRIPTION column when any listed signal has one.

signals send prints the one thing curl cannot tell you: whether your report opened a signal or folded into one that was already open. That is what makes it the right tool while you are wiring an integration up, even though curl is what runs in production.

The reads need a token with the read or signals scope; reporting, resolving and updating need signals. Deleting a signal and managing reporting URLs need your own login rather than a token - see API tokens and scopes.

An agent connected to DedNets can list your signals, read one in full, close one, report one, and set or clear a signal’s description (since v0.16.3). See Let an AI agent run DedNets.

Worth knowing before you ask an agent to test the wiring: signal_send posts a real occurrence and can reach your phone. Ask it to report to a throwaway key you can delete afterwards.

  • Reporting URLs are capped per account (50 by default). Delete one you no longer use to make room; the Console answers a create past the cap with a quota error. (Since v0.14.11.)
  • Signals are per account, not per reporting URL. Two machines reporting disk-full fold into one signal; put the machine in the key when that is not what you want.
  • There is no bulk resolve yet. Closing forty signals after one fix is forty clicks or forty CLI calls.
  • Quiet windows recur; there is no one-off snooze (“mute until 18:00 today”). Besides quiet windows, the only volume knob is the signals topic’s minimum priority.
  • Signals are kept for 30 days after their last activity, then removed automatically, whether they ended up resolved or not. “Last activity” means the most recent report, or the moment the signal resolved. A condition that keeps being reported is never old, however long ago it first fired, so an incident that has run for months is not at risk. A signal set to never auto-resolve is kept for as long as it is open. (Since v0.9.1. Before that, resolved signals were kept for 30 days and open ones were kept forever.)