Roadmap

I have a lot of features, ideas and the like I would like to implement. In short:

• Have the monitor worker run on all production servers (and on a schedule as backup) - done as of 2024-06-15
• Release for the open beta - done as of 2024-06-15
• Rework the monitor checks and comparison (mentioned in detail below) - done as of 2024-07-27
• Implement DNS monitor
• Implement DNS propagation monitor - done as of 2024-08-21
• Implement TLS/SSL monitor - done as of 2024-08-21
• Implement some kind of service up/down monitor tool - allowing 10 second checks on specific ports/protocols
• Up next Rework the emailing of notifications - should to work on a per-monitor level, instead of a global level
• Up next Work on company/team options, allowing for multiple users to work together on the same set of monitors (instead of shareing login credentials).
• Code and database cleanup (much need after several iterations)
• Handle monitor execution in case of failover events - done as of 2024-11-24
• Login in case of failover events and/or login from non-default servers (ensuring access to updated data, allowing update if settings while still keeping data consistency and the like) - done as of 2024-11-24

Tweaks / errors / issues / feedback

Backlog of issues to be fixed, and minor improvements:

• Monitor list on the frontpage need better sorting - done as of 2024-06-13
• System e-mails / notifications needs a brush up - done as of 2024-06-18
• Background worker needs better handeling of notifications sendout - done as of 2024-06-12
• Keep track of worker performance on multiple simultanious checks in progress - done as of 2024-07-27
• New users should be greeted in a way better way (unlike the empty page as of today) - done as of 2024-06-15
• Intros on the monitor edit pages - done as of 2024-06-15
• On settings of PortScan, it should a) validate the hostname, b) ask the user if multiple IP's is present on a given hostname - done as of 2024-07-04
• Monitor details - the list of log items: Should contain way more items, and details of each option should be availible in a popup. Should a graph of some kind be added?
• Make the api keys page responsive
• Make the settings page responsive
• Timestamp for calculation of api usage needs to be showen at the 'api keys' and 'settings' pages - done as of 2024-06-12
• Each user needs to be assigned to a specific production server, in order to ensure data consistency. Furthermore: *) Each monitor execution must be run from this assigned server *) Calculation of the api usage needs to be be done by each server, but limited to users assigned to this server *) Notification sendout needs to be done by each server as well - but limted to users assigned to this server - done as of 2024-06-13
• Monitor details should link to website with live run of tests - done as of 2024-08-03
• Optimize what data is saved on each check - removed unused parts of the JSON before saveing it to the DB - done as of 2024-06-16
• Keep track of any delays in monitor check executions - done as of 2024-06-16
• Backend page for monitoring users and monitors - done as of 2024-06-16
• When creating a monitor, the monitor-details page is more or less empty - it needs to show some usefull and informative info, while we wait for initial run of the monitor. - done as of 2024-06-17
• Alerting the user, X days before monitor expiration (only if it's a monitor running for several days). Alert on expiery as well.
• Introduce some level of discrete logging / finding - no need to tell the user about then 'several A/AAAA records' over and over. - done as of 2024-07-29
• Rework of the notification levels? Useing labels like 'warning' is not really describeing the issue. Consider the introduction of two new levels: Info and Warning - done as of 2024-08-03
• Rework of mails - could look better - done as of 2024-08-03
• WHOIS event info keep changeing - thus raising issues. Use a date only. - done as of 2024-07-29
• Portscan: Ensure any temporary fluxuations between open and closed, does not trigger any notifications. Only state changes between open and closed (or opposit) should trigger an alert/notification. - done as of 2024-07-29
• Remove 'last check state' label from the list of monitors - use the timetamp of last check, as an indication of last state as well. - done as of 2024-07-29
• Add fields for optional name and remarks on each monitor - it would make it easier to use and distinguish them. Having four monitors running on four IPs can easily turn out to be a problem, as you forget what each IP actually is. Thanks to David for the feedback. - done as of 2024-08-03
• TLS/SSL monitor: If running on a domain, covering several IP's with individual certificates, this will cause an endless strem of notifications, as the certificate will keep changeing. Similar problem on the DNS propagation monitor, if the DNS is doing geolocation work. How to handle these issues?
• TLS/SSL monitor: Tryout different approaches in order to achive more consistent results (less ports - higher timeout)
• TLS/SSL monitor: Change wording from 'CLOSED' to 'CLOSED/UNREACHABLE' (includeing all the implications of this)
• Rework the monitor edit-options - can be done, in a way more userfirendly manner
• DNS propagation: Propagation confirmation need to work in another way - in the final phase, there is a lot of uncertainties we need to adress - somehow. - done as of 2024-08-19
• DNS propagation: Is the final 'propagation complete' notification actually working as intended? - done as of 2024-08-19
• DNS propagation: Have the notifications tell, on auth changes, 1) what records were added, 2) what records were removed, and 3) what are the current set of records awaiting propagation - done as of 2024-08-19
• DNS propagation: For the final verification, use a timeframe that matches original ttl of the records / 2.
• DNS propagation: Include the record types in the alerts
• WHOIS monitor: Maybe ignore changes to eventinfo and/or persons, of they keep changeing for an extended period of time?
• Be more discrete about the first couple of monitor timeouts (no notifications) - done as of 2024-08-19
• Make it possible to add monitors in bulk (like 10 whois monitors) in one single go
• Check ratelimits upon monitor creation + other helper things
• Monitors details page could display current status better. Consider how to improve this.
• Retention strategy needs to be implemented - done as of 2024-07-27
• Housekeeping of disabled and expired monitors - done as of 2024-07-27
• Rework the monitor checks and comparison

  • The comparison needs to be way simpler and transparent
  • The check itself could be simplified as well
  • The principles/concepts of the different status'es and notifications needs to be simplified
  • Some level of confirmation of issues, queying from different servers, needs to be implementet

  - done as of 2024-07-27
• List of monitors (status) could display current status better. - done as of 2024-07-27
• Monitors details page could display current status better. Consider how to improve this.
• Remember the 'goto' url thrughout the process of creating a new user account - done as of 2024-09-12
• Gamify the post-signup process (for consideration)

Changelog

2024-11-24:

• 🏆 I've finally reached a point where monitor execution failover is actually working. It's been quite a project, with several iterations, and along the way, the need for secondary systems suddenly became apparent.
  
  In case of server failure or performance degradation, monitor execution is automatically failed over within minutes. Once the server is back, monitor execution is restored to its original state.
  
  It works as intended—I've done several tests and will continue to do so to ensure flawless execution, even when bad things happen.

2024-10-29:

• Over the last couple of weeks, I’ve been addressing the aftermath of the infamous server migration crash in early October - hence the missing progress on the changelog.
  
  While server downtime is expected, minimizing its impact is critical. As a result, I’ve been implementing failover handling at all levels, though this remains an ongoing process.
  
  Website and email access are covered, but there’s still work to do on the user and monitoring system, specifically to (1) ensure monitors execute smoothly even if a server goes offline and (2) handle login functionality during failover events.

2024-09-12:

• Remember the 'goto' url thrughout the process of creating a new user account

2024-09-01:

• TLS/SSL monitor would faild hard, if checking non SSL/TLS enabled website - has been fixed.

2024-08-30:

• DNS lookup issues on setup of the TLS/SSL monitor and portscan monitor has been solved.

2024-08-22:

• Improved the login page - better overview and readability.

2024-08-21:

• 🏆 Two new monitors are now available for testing:
  • TLS/SSL Certificate Monitor: This will keep track of changes, upcoming certificate expirations, and renewals. Never miss a TLS certificate expiration again.
  • DNS Propagation Monitor: This monitor will track DNS records of the primary/authoritative nameserver and alert you in case of changes. It will also monitor the subsequent propagation of the records. Let the monitor handle the tracking instead of doing endless and repetitive DNS lookups during the propagation period - as a first-ever, this monitor offers a 10-second check frequency.

2024-08-19:

• Several improvements on the DNS propagation monitor, includeing *) details of what changed at the SOA DNS server, *) better wrap up of the propagation process (waiting for confirm of full propagation), and *) better notifications
• The initial three errors - regardless of why - is way more discrete than previous

2024-08-18:

• This entry will cover work from the last couple of weeks
• Been working on the new TLS/SSL monitor and the new DNS propagation monitor. Both monitors are now in final testing before beeing made availible for public beta test within the next week. Among other things, the DNS propagation will offer 10 second check interval.
• Several minor tweaks has been made for the API - includeing a bugfixes, for the rate limit calculations
• An API endpoint has been made available for the DNS propagation tool

2024-08-04:

• Improveing the backend admin
• Initial work of a TLS/SSL monitor

2024-08-03:

• 'Last event' portion of details page, has been optimized for better overview - and more details.
• The notification levels has been reworked, now matching the levels of each monitor check. A couple of new intermediate levels has been introduced as well.
• Mail layout of each notification has been improved.
• An optional name has been implementet for all monitors. Host (or whatever the monitor is looking at) will be used per default - but if a name is present, it will be used to identify the monitor. Create or edit the monitor to input the name. Thanks to David for the feedback.
• Link for a live website run of the monitor, has been added to the details page of each monitor

2024-08-01:

• Minor issues about updateing each monitor has been solved (basic concurrency issue)

2024-07-31:

• Further work on the portscan and the detection/confirmation af changes
• Further work on the whois check of events (turned out to be a timezone issue causeing the issues - same date was parsed as one date in the US, and as antoher in Europe - UTC has been forced upon these checks).
• Invite to beta / call for feedback, among all newsletter subscribers

2024-07-29:

• Eventinfo of the WHOIS checks kept changeing - solved by using date without specific hours/minut/second info
• Reporting of Portscan changes has been made more stable, by tracking current state and last confirmed state - and only triggre notifications from one confirmed state to another. This means, that any temporary changes (for instance from 1: confirmed open, to 2: unformed closed, 3: unconfirmed open and then 4: confirmed open) do not trigger either alerts or notifications.
• Timestamp of last check on the list of monitors, is made to contain last status as well (color + hover title).
• A more discrete log level has been introduced, in order to keep track of important info - but wich doesn't make sense to put on display over and over again (for instance - the 'several A/AAAA records' is important, but does only make sense first time).

2024-07-27:

• 🏆 Remake of entire monitor system has been release. This includes a ton of both new and improved features. Been a bit of a struggle, but did manage to grind my way thrugh it :) The revised engine includes:
  • The list of monitors now include a status of each monitor, provideing an instant overview
  • The details of each monitor now include more data and a more userfiedly overview of status - both in general, and when looking into the specific details.
  • Check and comparision of the different monitors are simplified signigificantly.
  • Changes now need to be discovered by at least two different servers, before beeing considered as confirmed.
  • Retention strategy and housekeeping are implementet - disabled and expired monitors will be deleted in 45 days, and log items has a cutoff of 45 days.

2024-07-04:

• Been struggling quite a bit the last couple of weeks to make the monitoring work better and report in a more operational and usable manner. The current monitoring works fine, but I would like it to work even better. To achieve this, a complete remake of the monitoring system is needed. This may take several weeks of work (beside my day job) before I'm able to release this new and improved monitoring. Just a remark to explain why there may be less frequent updates in the upcoming weeks.
• Settings of the PortScan-monitor has been changed a bit, in order to handle any DNS-situations that could cause issues (like no records or multiple records).

2024-06-18:

• Performance optimizations on the underlying database
• Notification emails (and validation emails) has been revisited: *) ensuring proper formatting in darkmode *) proper linking and *) more operational structure of the informations within the e-mail.

2024-06-17:

• Some bugfixing on the monitor comparison. Removeing unused JSON parts (done yesterday), caused some issues when looking for closed ports.
• When disableing a user, all his/hers monitors are now disabled as well.
• When re-enabeling a user, the api key for the monitors, is re-enabled as well.
• If a user has selected grouping of notifications (for instance once every hour), a minor delay of 5 minutes has been implemented, if only a single noification is to be send - this allows for any additional changes after the initial change, to be a part of the mail as well (if a server goes offline for 2 minutes for instance - then the mail will include both DOWN message, and the following UP message).
• Once a monitor has been created, the intial status page of the monitor was very empty - now includes a 'first scan is imminent' message + auto refresh.

2024-06-16:

• Backend page for monitoring the healt of the user system has been added
• Monitor check delays are now tracked (difference between scheduled runtime vs actual runtime)
• Unused parts of the JSON is removed before saveing to the database - reducing the average byte size from about +50k to 1.5k.

2024-06-15:

• Better greeting for new users
• Meta on each type of monitor has been implemented - allowing better descriptions of monitor types and so forth
• The monitor worker, has been put into production on all servers
• Login-link added to menu
• 🏆 First release (yay!)

2024-06-13:

• User server assignment is introduced (and server assignment per monitor is removed)
• Monitor worker process, notification sendout, and housekeeping, now works on users assigned to current server only
• Better sorting of monitor frontpage has been implemented

2024-06-12:

• Performance improvements made on worker process for the monitors
• Better sendout of notifications implemented
• Another iteration on PortScan open/close confirmation is put in production
• Page for roadmap / changelog added
• Timestamps of api usage calculations has been added to the 'settings' and 'api keys' pages