The background for this is in the announcement on the blog. Here’s the nitty gritty of what we’re (eventually) implementing! [1]
One of the tricky things about this change is that everyone is very used to midnight (or 3am) and so — even if you opt in to a different deadline — it has to be kept very front-and-center. Next to the countdown timer (somewhere in the box with it) will be shown the time of day of the deadline. When we first make this live, that time of day may need to be especially prominent. We do still want the countdown itself even though it’s redundant information because we want to convey that it’s dynamic and to have that ticking time bomb feel. But the time of day deadline could be big and the countdown small. (Before going live we could make the time of day small and keep the countdown big and then swap them as part of officially launching.)
How do we distinguish between someone who wants a 6am end-of-day because they want to be able to pull a beemergency all-nighter and someone who wants a 6am end-of-day because they want to be forced to dispatch their beemergencies before breakfast? They’re both choosing 6am but they mean quite different things.
It will be made obvious by picking the deadline via a slider:
[6am ---- 9am ---- 12pm ---- 3pm ---- 6pm ---- 9pm ---- 12am ---- 3am ---- 6am]
If you slide past midnight then a little footnote appears below the slider:
(Night-owl mode! After midnight so technically the next day, which is fine.)
If you slide before noon then the footnote is:
(Earlybird mode! Dispatch your beemergencies before breakfast.)
Webcopy for confirmation of a deadline before midnight (or maybe just an FAQ unless we can make it more concise than this):
WARNING: A deadline before midnight counts as the day boundary, even if it’s in the morning. If you manually add data for a given day then we respect the date you specified (though if it’s an emergency day and the graph has already derailed then it’s too late, the data’s accepted but won’t undo the derailment). If autodata comes in after the deadline then it’s displayed and plotted for the following day.
NB: To simplify the initial implementation we’re not going to allow earlybird deadlines before noon.
A tricky thing about choosing your deadline is preventing you from pushing the deadline back as it approaches, defeating the whole point. Akrasia-proofing it would be fine if that’s what’s easiest to implement (we need a generalized akrasia-proofer for various fields anyway) but it’s harsher than necessary. But overharshness — having a suboptimal deadline for a week — is not the end of the world. Even if you had an earlybird deadline when you wanted a night-owl deadline, just remember to think of orange days as your beemergency days for a week. [2]
We treat the deadline as the day boundary, meaning that datapoints after midnight but before a night-owl deadline count as what’s technically the previous day. Similarly, but potentially more confusingly, datapoints after an earlybird deadline count as the following day. That may sound ugly but it doesn’t violate the sanctity of data to the point of actually changing timestamps. The fudging is just in how datapoints are displayed and plotted.
If you have a night-owl deadline then datapoints timestamped after midnight but before the deadline are treated as if timestamped for the previous day. They’ll display that way in recent data and plot that way on the graph.
Technical Note: The deadline is stored as a number of seconds after midnight, which is a negative number for earlybird deadlines. For example, a night-owl deadline of 3am would be \(3\cdot60^2\), a standard midnight deadline would be 0, and an earlybird deadline of 11pm would be \(-1\cdot60^2 = -3600\). A reasonable range for deadlines is \(-18\cdot60^2\) to \(6\cdot60^2\) (6am earlybird to 6am night-owl).
Given a deadline, \(d\), defined like that, compute the daystamp (year-month-day string) for a datapoint \((t, v)\) by converting unixtime $$t - d$$ to year-month-day in the user’s timezone. The formula is general, applying to both earlybird and night-owl deadlines. For example, with an earlybird deadline of 9am and a datapoint on the 30th, the daystamp is the 30th until 9am, and the 31st — though that’s blatantly “tomorrow” — thereafter.
This is very much what people expect in night-owl mode (cf “it’s still the 30th to me!” and “I want to define when my day ends!”). It’s less expected in the earlybird case but at least it’s consistent. And, again, the timestamp is never altered, we just use \(t-d\) (per technical note above) for turning the timestamp into a daystamp (year-month-day string).
Carets simply expand to Right Now (or shifted by a multiple of 86400 in the case of multiple carets) and thus work just like autodata datapoints. This fixes the longstanding confusion with having to submit a double caret after midnight.
Any datapoint given manually with an explicit day of month gets parsed to the dayfloored [TODO] timestamp (the earliest possible time on that day) and then has the deadline value added. Since all timestamps have the deadline value subtracted for the purposes of displaying and graphing, this ensures that a datapoint entered as, say, “the 30th” always displays and graphs as the 30th.
Scenario: You have an eep day on the 30th with a 9am earlybird deadline and you miss it. You derail and get rerailed. Then you add a datapoint of 30 1 which is timestamped as 9am on the 29th, right after yesterday’s deadline so is shown to the user as the 30th, as they expect. If the derailment wasn’t legit it could be undone and the data would be enough to keep you on track. That’s the same as the status quo where you forget to enter data in time but enter it retroactively.
The only weirdness here is that we say, e.g., “Today (30)”.
Case 1: For night owls the word “Today” gets this footnote during the night-owl zone (midnight till the deadline):
It’s after midnight but it still counts as the [30th] till the [3am] [deadline](link to settings).
Case 2: For night owls when you choose “Yesterday (29)” during the night-owl zone there’s no footnote (too convoluted to explain that it’s technically the day before yesterday and it’s not so critical to distinguish those anyway).
Case 3: For earlybirds during the earlybird zone, “Today (30)” turns into “Tomorrow (31)” and gets this footnote during the earlybird zone (deadline till midnight):
It’s after the [5pm] deadline so it counts as the [31st] now.
Case 4: For earlybirds, “Yesterday (29)” turns into “Today (30)” at the deadline and gets this footnote:
It’s after the [5pm] deadline so it counts as the [31st] now but you can still enter retroactive data.
Actual midnight is irrelevant. Refreshes happen when you hit refresh, before sending reminders (including zeno reminders), and at the deadline. (For some autodata sources, new data is pushed to Beeminder which also triggers a graph refresh; for others, Beeminder pulls new data when refreshing.)
Beebrain only knows about the daystamps so, for example, if you have datapoints today that are both before and after your earlybird deadline, then
Beebrain will plot the graph with asof equal to tomorrow’s daystamp, which is consistent with thinking of datapoints after the deadline as counting for tomorrow.
Scenario: If your graph is currently orange and a refresh is happening now at the 9am earlybird deadline (or one second after) then the :asof will be tomorrow’s daystamp and the graph will refresh as red. And 24 hours later it will derail.
We don’t bump the API version, we simply fix the unixtimes to be correct [4], and include an additional daystamp field (eg, “20140831”) along with the unixtimes. We straw-polled users of the API and the consensus was that we should just fix the timestamps in the API and not worry about backward compatibility with the previous craziness.
Technical note: We use daystamps like “20140831” instead of “2014-08-31” not just to shave off storage/bandwidth cost but to avoid confusion that might ensue if you left off preceding zeros, like “2014-8-31”. The condensed version is ascii-sortable as well as numerically sortable. And it avoids the question of what delimiters are allowed.
See also: Continuous Beeminding.
[1] Crazy alternative proposal not currently on the table: There’s no such thing as “6am the next morning”. You can set your deadline to be 6am but that would mean bright and early on the emergency day. Night owls would then have to be aware that the real emergency day — when they can pull the all-nighter to get back to safety — is when they’re in the orange. With more flexible zeno polling and better use of the panic threshold that might be tenable. Right now everything is very eep-day-focused and it would be too confusing.
More flexible zeno polling means starting on the previous day. Otherwise a night owl who wanted, say, a 3am deadline would not be able to have Beeminder tell them to panic until 3 hours before derailment. But the generalization is actually pretty natural as well: You pick a time of day for zeno start time and if it’s after your deadline then it takes it to mean the previous day at that time. (That also means you could have zeno polling start up to 24 hours before the deadline. If the deadline is, say, 5pm, just make the zeno start time be 5:01pm.)
[2] But if it’s not too much of a special case, here’s an alternate spec for changing the deadline tomorrow:
Ideally any changes to the deadline simply take affect the next day. To pin that down, if the current deadline and the new deadline are both 12 hours [3] or more in the future then just allow the change immediately. If either the current or the new deadline are less than 12 hours in the future then still allow any change, but display a warning:
(The new deadline of XX:XX will take effect after the upcoming YY:YY deadline.)
For example, imagine it’s 9am and today’s deadline is currently 10pm. I can change it to 9pm and the change will be immediate. If I try to change it to anything earlier than 9pm then then new deadline won’t take effect till the next day. And if it’s after 10am with a 10pm deadline then I can’t make any immediate changes to the deadline, including pushing it further out.
[3] Why 12 hours instead of 24? Because if you just passed the deadline and thought it sucked you should be able to change tomorrow’s deadline. But since you just passed today’s deadline, tomorrow’s deadline is within 24 hours. So we need a window and the simplest one is a full 12 hours.
[4] This was so WTF maybe we won’t even explain it, but this is a placeholder in case we decide to… But the important thing is that the long national nightmare is finally over!
We had an interesting debate between the above worse-is-better choice where we minimize complications and special cases by just always treating the deadline as the day boundary, and The Right Thing which would be to respect the sanctity of the data and never mislead users about what happened when. For posterity, this was the tentative alternative spec.
We need to make a strong distinction between a datapoint on the 31st before the deadline and a datapoint on the 31st after the deadline. Users can’t be expected to understand that a datapoint that actually happened on one day is recorded as if it happened the following day just because it was after the deadline. The deadline is purely an aspect of the commitment device and can’t impact the QS aspect. Data happens when it happens and can’t be fudged.
Except night-owl deadlines. In that case it’s still the previous day until the deadline, even if that’s the next morning. Datapoints are recorded and plotted that way since that’s what users expect. Only for earlybird deadlines (before midnight) do we need a special case for plotting and recording the datapoint on the day it happened but marking it as having missed the deadline.
The distinction will be made visually on the graph and in the list of datapoints, like having a sad-face icon by each point that missed the deadline for that day.
Commentary: The unfortunate inconsistency here is how we have to treat the earlybird and night-owl cases differently. Night-owl goals have datapoints bucketed from today’s deadline to tomorrow’s deadline, while earlybird goals have datapoints bucketed from midnight to midnight. The special cases and implementation difficulties are in how we distinguish for the user between datapoints before and after the earlybird deadline on a given day.
PS: Or maybe everything could be bucketed midnight-to-midnight and datapoints that miss an earlybird deadline get a sadface and datapoints that are after midnight but before the nightowl deadline get happyfaces. So a sadface means that it was too late even though the date is ostensibly correct. And a happyface means that the datapoint counted for the previous day even though the date is ostensibly wrong. It’s not necessarily crazy but I’m pretty sure it’s messier than the new status quo of treating anything before the deadline as the nominal day and anything after the deadline as being the next day, no matter when the deadline is.